12 expert agents with 200+ skills. Running 24/7 on local compute. Built for South African business.
Each agent is grounded in domain expertise — not generic chat. They know SA regulations, local suppliers, and your market context.
Real deliverables generated by AI agents with skill-grounded expertise. Not generic advice — actionable work product.
## 1️⃣ Architecture Overview | Layer | Technology | Why it fits | South‑Africa note | |-------|------------|-------------|-------------------| | **Source** | PostgreSQL (3 yrs, 50M rows) | Native ...
Prompt: We have 3 years of transactional data in PostgreSQL (50M rows). Design a data warehouse architecture for customer LTV analysis, churn prediction, and cohort analysis. Include the dimensional model with specific fact and dimension tables.
| Layer | Technology | Why it fits | South‑Africa note |
|-------|------------|-------------|-------------------|
| Source | PostgreSQL (3 yrs, 50M rows) | Native source – transactional logs are already in a relational format. | ✅ |
| Ingestion / Raw lake | Airbyte → S3 (af‑south‑1) (or DMS for CDC) | Airbyte has a PostgreSQL connector that can run incremental pulls every night and dump them as Parquet files partitioned by event_date. DMS gives true CDC but is more expensive. | All objects stored in an af‑south‑1 bucket – meets POPIA “data residency” requirement. |
| Metadata catalog | AWS Glue Data Catalog (linked to S3) | Lets Snowflake read the raw files with a source table (raw.<schema>.<table>). | ✅ |
| Warehouse | Snowflake on af‑south‑1 | • SQL‑first, ACID transactions. <br>• Native clustering / partitioning → very fast aggregations for LTV and cohort work. <br>• Built‑in Snowpark for any needed PySpark transformations. | All compute & storage stay inside the region. |
| Transformations | dbt (Snowflake adapter) | Declarative data model, lineage, tests, documentation. <br>Runs nightly jobs from dbt Cloud or GitHub Actions. | ✅ |
| Feature store / ML ingestion | Feast + Snowflake (or SageMaker Feature Store) | For churn‑prediction you need a snapshot of customer features that can be served to your model in real time. <br>Feast integrates with Snowflake, giving a single source for the training data and online feature store. | ✅ |
| Analytics & BI | Looker Studio (free) → Snowflake OR Power BI + Snowflake connector | Executive dashboards (LTV, churn trend, cohort curves). <br>Looker Studio is low‑cost and integrates out of the box with Snowflake. | ✅ |
| Model training / ops | SageMaker or Vertex AI (if you prefer GCP) – or custom Jupyter on EC2/ECS | Trains churn models using the snapshot from Feast. <br>Training data lives in Snowflake → exported as Parquet to S3; SageMaker pulls it via a role. | If you want everything inside AWS, use SageMaker. |
Data flow diagram (text)
``
┌───────────────────────┐
│ PostgreSQL (source) │
└─────────▲─────────────┘
│
Airbyte CDC / Incremental Pull
▼
┌───────────────────────┐ ───► Snowflake raw schema
│ S3 bucket (Parquet) │
└─────────▲─────────────┘
│
Glue Data Catalog
▼
┌───────────────────────┐ ───► Snowflake staging & marts
│ dbt projects │
└─────────▲─────────────┘
│
Snowflake warehouse (Warehouse + Feature Store)
▼
┌───────────────────────┬───────────────────────┐
│ BI / Dashboards │ ML training │
│ Looker Studio / Power│ SageMaker / Feast │
└───────────────────────┴───────────────────────┘
`
---
Below is a star schema that covers LTV, churn and cohort work.
All dimension tables use surrogate keys (*_SK); the fact tables reference them.
| Table | Granularity | Key fields | Notes |
|-------|-------------|------------|-------|
| dim_customer | 1 row per customer | customer_sk, customer_id, first_name, last_name, email, created_at, status (ACTIVE / INACTIVE), segment, signup_channel | Keeps a stable lookup for all facts. |
| dim_product | 1 row per product SKU | product_sk, sku, name, category, price, brand | Add if you want spend by product or category. |
| dim_time | 1 day row | time_sk, date, day_of_week, week, month, quarter, year | Drives all time‑series aggregations. |
| dim_region | 1 row per region | region_sk, country, state, city | Useful for geography‑driven churn signals. |
| dim_channel | 1 row per acquisition channel | channel_sk, channel_name, source_type | E.g. FACEBOOK, ORGANIC, EMAIL. |
| Table | Purpose | Key fields (FK) | Measures | Notes |
|-------|---------|-----------------|----------|-------|
| fact_sales | Individual transaction (used for LTV & churn metrics) | sale_sk, time_sk, customer_sk, product_sk, region_sk | quantity, unit_price, discount_amount, net_revenue | Partition by time_sk; cluster on customer_sk. |
| fact_customer_activity | Non‑transactional events (logins, support tickets) | activity_sk, time_sk, customer_sk | activity_type enum (LOGIN, SUPPORT_TICKET, WEB_SESSION), count | Enables churn features like recency and engagement. |
| fact_ltv | Pre‑aggregated LTV per customer & time slice (monthly) | ltv_sk, customer_sk, time_sk | cumulative_revenue | Incremental dbt model; refreshed nightly. |
| Table | Purpose | Key fields |
|-------|---------|------------|
| snapshot_customer_features | Daily snapshot of RFM, last‑active date, support tickets count etc. | customer_sk, snapshot_date | Materialised as a dbt incremental model; feeds Feast or your ML training data store. |
> Why both fact_sales & fact_ltv?
> fact_sales keeps the atomic level for audit & detailed queries (e.g., “show me all purchases by customer X in June”).
> fact_ltv gives a ready‑to‑use view that is far cheaper to query when you just need the cumulative value per month.
---
`sql
-- 1. Dimensions -------------------------------------------------
CREATE TABLE dim_customer (
customer_sk NUMBER AUTOINCREMENT,
customer_id VARCHAR(50) NOT NULL, -- natural key
first_name STRING,
last_name STRING,
email STRING,
created_at TIMESTAMP_LTZ,
status STRING, -- ACTIVE / INACTIVE
segment STRING,
signup_channel STRING,
PRIMARY KEY (customer_sk),
UNIQUE (customer_id)
);
CREATE TABLE dim_product (
product_sk NUMBER AUTOINCREMENT,
sku VARCHAR(50) NOT NULL,
name STRING,
category STRING,
price FLOAT,
brand STRING,
PRIMARY KEY (product_sk),
UNIQUE (sku)
);
CREATE TABLE dim_time (
time_sk NUMBER AUTOINCREMENT,
date DATE NOT NULL,
day_of_week VARCHAR(10),
week INTEGER,
month INTEGER,
quarter INTEGER,
year INTEGER,
PRIMARY KEY (time_sk),
UNIQUE (date)
);
-- 2. Facts -------------------------------------------------------
CREATE TABLE fact_sales (
sale_sk NUMBER AUTOINCREMENT,
time_sk NUMBER NOT NULL, -- FK to dim_time
customer_sk NUMBER NOT NULL, -- FK to dim_customer
product_sk NUMBER NOT NULL, -- FK to dim_product
region_sk NUMBER, -- optional FK to dim_region
quantity INTEGER,
unit_price FLOAT,
discount_amount FLOAT,
net_revenue FLOAT,
PRIMARY KEY (sale_sk),
INDEX idx_sales_date (time_sk)
);
-- cluster on customer for LTV scans
ALTER TABLE fact_sales CLUSTER BY (customer_sk);
CREATE TABLE fact_customer_activity (
activity_sk NUMBER AUTOINCREMENT,
time_sk NUMBER NOT NULL,
customer_sk NUMBER NOT NULL,
activity_type STRING, -- LOGIN / SUPPORT_TICKET ...
count INTEGER DEFAULT 1,
PRIMARY KEY (activity_sk)
);
CREATE TABLE fact_ltv (
ltv_sk NUMBER AUTOINCREMENT,
time_sk NUMBER NOT NULL,
customer_sk NUMBER NOT NULL,
cumulative_revenue FLOAT,
PRIMARY KEY (ltv_sk)
);
`
> Tip: In Snowflake you can create view vw_customer_ltv_monthly that joins fact_ltv, dim_customer, dim_time. This keeps your dashboards lean.
---
).E.g. stg_transactions (raw Parquet from Airbyte).
E.g. int_sales_cleaned.
, dim_product, …).Model:
`sql
-- models/marts/core/dim_customer.sql
{{ config(materialized='table') }}
SELECT
ROW_NUMBER() OVER (ORDER BY customer_id) AS customer_sk,
customer_id,
first_name,
last_name,
email,
created_at,
status,
segment,
signup_channel
FROM {{ ref('stg_customers') }}
`
`sql
-- models/facts/fact_sales.sql
{{ config(materialized='incremental', unique_key='sale_id') }}
SELECT
ROW_NUMBER() OVER (ORDER BY sale_id) AS sale_sk,
DATE_TRUNC('day', transaction_ts)::DATE as txn_day, -- will join to dim_time
customer_sk,
product_sk,
region_sk,
quantity,
unit_price,
discount_amount,
net_revenue
FROM {{ ref('int_sales_cleaned') }} s
JOIN {{ ref('dim_customer') }} c USING (customer_id)
JOIN {{ ref('dim_product') }} p USING (sku)
LEFT JOIN {{ ref('dim_region') }} r USING (region_code)
WHERE NOT EXISTS (
SELECT 1 FROM fact_sales fs WHERE fs.sale_id = s.sale_id
)
`
`sql
-- models/snapshots/customer_features.sql
{{ config(materialized='incremental', unique_key=['customer_sk','snapshot_date']) }}
WITH recency AS (
SELECT customer_sk,
MAX(transaction_ts) AS last_purchase_ts
FROM {{ ref('fact_sales') }}
GROUP BY customer_sk
),
frequency AS (
SELECT customer_sk,
COUNT(DISTINCT DATE_TRUNC('day', transaction_ts)) AS purchase_days
FROM {{ ref('fact_sales') }}
GROUP BY customer_sk
)
SELECT c.customer_sk,
CURRENT_DATE() AS snapshot_date,
COALESCE(r.last_purchase_ts, '1970-01-01')::DATE as last_purchase_date,
f.purchase_days,
(SELECT COUNT(*) FROM {{ ref('fact_customer_activity') }} a
WHERE a.customer_sk = c.customer_sk AND a.activity_type='LOGIN')
AS daily_logins_last_30d
FROM {{ ref('dim_customer') }} c
LEFT JOIN recency r USING (customer_sk)
LEFT JOIN frequency f USING (customer_sk);
`
, unique, custom test for monotonic LTV growth).`yaml
models:
columns:
tests:
`
---
) into Feast’s online store.).---
| Area | Tool / Practice |
|------|-----------------|
| Data residency | All S3, Snowflake, Glue in af-south‑1. |
| Access control | Snowflake roles: SYSADMIN, DATAENGINEER, BI_READER, ML_OPERATOR. Least privilege on every table. |
| Audit logs | CloudTrail + Snowflake query history → detect unauthorized reads/writes. |
| Data quality | dbt tests, Great Expectations for raw lake. |
| Metadata catalog | AWS Glue (linked to Snowflake) + dbt docs. |
| PII handling | Store PII only in dim_customer; encrypt all columns with dynamic data masking; anonymise logs before moving to the feature store. |
---
. – schedule nightly., dbt test. and fact_ tables; publish to executive users.---
, dim_product, dim_time, dim_region + three fact tables (fact_sales, fact_customer_activity, fact_ltv) gives you everything you need for LTV, churn & cohort analysis.Feel free to tweak sizes (warehouse size, clustering columns) once you start seeing actual query performance numbers. The above design is the minimum viable architecture that scales cleanly as your data volume grows beyond 50 M rows.
Below is a **minimal yet production‑ready** layout for an e‑commerce analytics project that uses the classic **staging → intermediate → mart** pattern. All files are written with dbt v1.x in mind an...
Prompt: Set up a dbt project structure for our e-commerce analytics. We need staging, intermediate, and mart layers. Show me the folder structure, YAML config, and SQL for a customer lifetime value mart model.
Below is a minimal yet production‑ready layout for an e‑commerce analytics project that uses the classic staging → intermediate → mart pattern.
All files are written with dbt v1.x in mind and are ready to drop into a Snowflake/BigQuery/PostgreSQL data warehouse – just change the database/schema values in your profiles.yml.
> ⚡️ Quick‑start tip: create a new repo, run dbt init <project-name> and copy the snippets below over.
> 📦 Keep the root folder named after your project (e.g., ecom_analytics).
---
``
ecom_analytics/
├── models/
│ ├── staging/ # raw → cleaned, consistent naming
│ │ ├── stg_customers.sql
│ │ ├── stg_orders.sql
│ │ ├── stg_order_items.sql
│ │ └── stg_products.sql
│ ├── intermediate/ # business logic & derived facts
│ │ ├── int_customer_summary.sql
│ │ └── int_order_revenue.sql
│ ├── marts/ # end‑user consumption models
│ │ └── mart_customer_lifetime_value.sql
│ ├── sources.yml # source definitions for all tables
│ └── mart_customer_lifetime_value.yml # column docs/tests
├── macros/
├── tests/
└── dbt_project.yml
`
Tip: If you have many marts, split them into sub‑folders (marts/sales/, marts/marketing/…) to keep the top level tidy.
---
`yaml
version: 2
sources:
database: {{ target.database }}
schema: stg_raw
tables:
description: "Raw customer records from the CRM"
description: "Orders placed by customers"
description: "Line‑items belonging to each order"
description: "Product catalog"
`
> Why a source file?
> 1️⃣ Keeps the lineage clean.
> 2️⃣ Enables source freshness and column tests out of the box.
---
`yaml
version: 2
models:
description: |
Customer Lifetime Value (CLV) aggregated at the customer level.
Calculated as the sum of all historical order revenue,
optionally adjusted for retention and frequency in downstream models.
columns:
tests:
description: "Earliest purchase date"
description: "Most recent purchase date"
description: "Count of all orders"
tests:
description: "Cumulative gross revenue (USD)"
tests:
description: "Total revenue divided by total orders"
description: "Average time between purchases"
`
> Testing strategy – we only test the schema here; business logic is validated in downstream dashboards.
---
(just enough to get started)`yaml
name: ecom_analytics
version: 1.0
profile: default # <-- use your own profile name
source-paths: ["models"]
analysis-paths: []
test-paths: ["tests"]
macro-paths: ["macros"]
data-paths: []
model-paths: ["models"]
materialized: view
`
> You can override per‑model in the SQL file; the default is view to keep the warehouse lean until you need a persisted table.
---
`sql
-- models/staging/stg_customers.sql
{{ config(materialized='table') }}
SELECT
customer_id,
email,
first_name,
last_name,
created_at::date as signup_date,
country_code,
-- standardise gender codes if present
CASE WHEN LOWER(gender) IN ('m', 'male') THEN 'M'
WHEN LOWER(gender) IN ('f', 'female') THEN 'F'
ELSE NULL END AS gender
FROM {{ source('raw', 'customers') }}
`
> Tip – keep all transformations that clean raw data (type casts, standardisation, de‑duplication) in staging.
> For performance, materialise as table (or incremental if your source updates frequently).
---
`sql
-- models/intermediate/int_order_revenue.sql
{{ config(materialized='view') }}
SELECT
o.order_id,
o.customer_id,
o.order_date,
SUM(oi.quantity * oi.unit_price) AS order_revenue,
SUM(oi.quantity) AS items_purchased
FROM {{ ref('stg_orders') }} o
JOIN {{ ref('stg_order_items') }} oi ON oi.order_id = o.order_id
GROUP BY 1,2,3
`
> This view gives us a clean “order‑level” fact that downstream marts can use.
---
`sql
{{ config(
materialized='incremental',
unique_key='customer_id'
) }}
WITH order_totals AS (
SELECT
customer_id,
MIN(order_date) AS first_order_date,
MAX(order_date) AS last_order_date,
COUNT(*) AS total_orders,
SUM(order_revenue) AS total_revenue
FROM {{ ref('int_order_revenue') }}
GROUP BY 1
),
-- Optional: compute days between orders for a simple frequency estimate
days_between AS (
SELECT
customer_id,
AVG(DATEDIFF(day, LAG(order_date) OVER (PARTITION BY customer_id ORDER BY order_date), order_date)) AS avg_days_between_orders
FROM {{ ref('int_order_revenue') }}
GROUP BY 1
)
SELECT
o.customer_id,
o.first_order_date,
o.last_order_date,
o.total_orders,
o.total_revenue,
ROUND(o.total_revenue / NULLIF(o.total_orders, 0), 2) AS avg_order_value,
d.avg_days_between_orders
FROM order_totals o
LEFT JOIN days_between d USING (customer_id)
{% if is_incremental() %}
WHERE EXISTS (
SELECT 1 FROM {{ ref('int_order_revenue') }} i
WHERE i.customer_id = o.customer_id
)
{% endif %}
`
| Step | Why it matters |
|------|-----------------|
| Order totals per customer | Core CLV driver – revenue & frequency. |
| First/last order dates | Gives you cohort and churn window insight. |
| Avg days between orders | Proxy for repeat‑purchase behaviour (can feed into more sophisticated retention models). |
| Incremental logic | Only reprocess customers who have new order data since the last run, keeping runtime fast even with millions of customers. |
> 📌 If you want a discounted CLV or a multi‑period prediction, plug that calculation in after order_totals.
---
`bash
dbt seed # if you have any CSVs
dbt run --models marts.* # build only the CTV mart (will implicitly build staging/intermediate first)
`
> Debugging tip – enable --profile-dir . to point dbt at your local profiles.yml.
---
| Want to… | Add / Change |
|----------|--------------|
| Capture seasonality or promotion impact on CLV | Create a int_order_metadata.sql that flags orders with coupons, sales events; join into CTV. |
| Expose the mart via BI (Looker/Tableau) | Materialise as a table (materialized='table') and set appropriate permissions in your warehouse. |
| Add predictive LTV | Create a new model mart_predicted_ltv.sql that imports a trained model from SageMaker/Vertex and scores it against the mart rows. |
| Enforce governance | Add data tests in models/mart_customer_lifetime_value.yml, enforce lineage via source freshness, or add a column‑level test for total_revenue >= 0. |
---
1️⃣ Folder layout – staging/ → intermediate/ → marts/.
2️⃣ YAML – source definitions + model docs/tests.
3️⃣ SQL – clean staging, business logic in intermediate, and a production‑ready incremental CLV mart.
Drop the code into your repo, adjust the schema/database names for your environment, run dbt build`, and you’ll have a reliable Customer Lifetime Value dataset that can be consumed by dashboards or fed to downstream ML pipelines. Happy modeling!
## 1️⃣ Business Context & Decision Goal | Question | Answer | |---------|--------| | **What business decision will the model inform?** | *“If we know which customers are likely to churn in the next...
Prompt: We want to predict customer churn for our SaaS platform. We have 18 months of usage data, payment history, and support tickets. Frame this as an ML problem: target variable, features, model selection, and how to measure success.
| Question | Answer |
|---------|--------|
| What business decision will the model inform? | “If we know which customers are likely to churn in the next month, we can launch a targeted win‑back campaign (discount, usage nudges, proactive support) and measure the incremental revenue retained.” |
| Why is this valuable? | • Churn cost ≈ 10–30 % of annual ARR for SaaS. <br>• A 5 % lift in retention could mean a multi‑million‑rand saving/earnings per year (South African market). |
---
| Name | Definition | Why this label? |
|------|------------|-----------------|
| churned_next_month | 1 if the customer does not make any payment / logs in during the next 30 days after a given “prediction date”; 0 otherwise. | • Aligns with marketing’s win‑back window.<br>• Keeps the problem strictly binary and time‑bound, avoiding future leakage (e.g., using data from after churn). |
Prediction horizon – we’ll predict churn for month + 1 relative to a “snapshot” date (e.g., the 1st of each month).
---
| Source | Raw Data | Typical Feature Ideas |
|--------|----------|-----------------------|
| Usage Logs | Event tables: page_view, feature_use, session_duration | • Recency (days since last session), <br>• Frequency (sessions per week/month), <br>• Intensity (avg. session length, number of feature X used), <br>• Trend (slope of usage over past 3 months) |
| Payment History | payments: date, amount, plan id, status | • Last payment amount, plan type<br>• Days since last paid invoice, overdue days<br>• Payment frequency consistency (e.g., monthly vs. irregular)<br>• Plan changes (upgrade/downgrade/expansion) |
| Support Tickets | tickets: open_date, close_date, priority, sentiment, category | • Ticket count per month<br>• Avg. resolution time<br>• Sentiment score (positive/neutral/negative)<br>• % of tickets flagged as “critical” or “re‑open” |
All features must be extracted only from data available before the prediction date to avoid leakage.
---
| Domain | Core Feature Group | Implementation Hint |
|--------|--------------------|---------------------|
| Usage | last_session_date, sessions_per_week, avg_session_len, feature_x_usage_rate | Use windowed aggregates (MAX, COUNT, AVG) over the prior 30–90 days. |
| Trend | usage_slope_3m, payment_amount_change | Fit a simple linear regression per customer (or use lead/lag differences) and store slope as a feature. |
| Payment | last_payment_date, days_since_last_payment, is_overdue, plan_id | Convert plan IDs to categorical embeddings or one‑hot vectors. |
| Support | tickets_last_3m, avg_resolution_time, sentiment_score_avg | If sentiment scores not available, use a lightweight NLP model (e.g., DistilBERT) on the ticket body. |
Feature Storage – put all engineered columns into a staging or mart table (customer_features) that is refreshed nightly.
---
| Split | Range | Reason |
|-------|-------|--------|
| Train | Jan 2024 – Apr 2024 | Historical period; enough data for learning. |
| Validation (tune) | May 2024 | Recent but still unseen; protects against overfitting to last month’s patterns. |
| Test (final) / Deployment | Jun 2024 | Mimics production churn horizon: we predict June‑churn on features computed from Jan–May data. |
Avoid shuffling. Use chronological splits or sliding windows if you need a larger validation set.
---
| Stage | Models | Why |
|-------|--------|-----|
| Baseline | Logistic Regression (with L2 penalty) | Fast, interpretable, baseline performance. |
| Mid‑tier | Gradient Boosting: LightGBM or XGBoost | Handles non‑linearities & missing values well; state of the art for tabular data. |
| Advanced | CatBoost / GBDT tuned with Bayesian optimisation | Handles categorical variables natively; often better on sparse categories. |
Pipeline:
---
| Technique | How to apply |
|-----------|--------------|
| Resampling – SMOTE, Random Oversample / Undersample | Apply only to training set; keep validation/test untouched. |
| Class weights – scale_pos_weight in XGBoost / class_weight='balanced' | Reflects relative cost of missing a churner vs false alarm. |
| Threshold tuning – ROC‑curve search for point maximizing business KPI (e.g., lift on top 10 % cohort). | After model training, use validation set to pick probability threshold that maximises expected retained revenue. |
---
| Metric | Why |
|--------|-----|
| ROC‑AUC | Overall ranking quality; insensitive to class imbalance. |
| PR‑AUC | Focuses on minority (churn) performance; useful when churn rate < 10 %. |
| Lift at top decile / 5 % | Business relevance: how many true churners are captured in the most targeted group. |
| KPI | How to calculate | Target (example) |
|-----|------------------|-----------------|
| Revenue Retained per 1000 prospects | (Average Revenue Per Customer) × Lift × 1000 | > R15 000 |
| Cost per True Positive | Marketing Spend / # of true churners caught | < R3 000 |
| Reduction in Churn Rate | Baseline churn – model‑guided churn | ≥ 1.5 % absolute drop |
Always present both sides. Technical metrics justify the model, business metrics convince stakeholders.
---
| Step | Action | Tool |
|------|--------|-----|
| Data pipeline | Ingest logs → raw S3 (or Cloudflare D1) → nightly ELT via dbt | AWS Glue / Airbyte + dbt |
| Feature store | Persist engineered features in a warehouse (Snowflake/BigQuery) for training and serving | Feast (open‑source) or Vertex AI Feature Store |
| Model registry | Track each trained artifact, hyperparameters, performance | MLflow Model Registry |
| Serving pattern | Batch scoring daily to populate churn_score table → used by marketing automation | dbt + scheduled Airflow DAG |
| Monitoring | PSI on key features; drift alerts; AUC trend in Grafana | S3/CloudWatch logs → Prometheus/Grafana |
| Retraining trigger | If PSI > 0.2 or AUC drops 5 % → run nightly retrain pipeline | Prefect / Airflow DAG |
---
churned_next_month.With these pieces in place, you’ll have a repeatable churn‑prediction workflow that turns raw usage logs into actionable insights and measurable financial impact for your SaaS platform.
A chatbot answers questions. An agent does work. The difference is defined skills, persistent memory, and tool access.
Browse real deliverables from each agent — marketing plans, financial models, legal analysis, IoT specifications, and more.
View Showcase