Prompt: We have 3 years of transactional data in PostgreSQL (50M rows). Design a data warehouse architecture for customer LTV analysis, churn prediction, and cohort analysis. Include the dimensional model with specific fact and dimension tables.
1️⃣ Architecture Overview
| Layer | Technology | Why it fits | South‑Africa note |
|-------|------------|-------------|-------------------|
| Source | PostgreSQL (3 yrs, 50M rows) | Native source – transactional logs are already in a relational format. | ✅ |
| Ingestion / Raw lake | Airbyte → S3 (af‑south‑1) (or DMS for CDC) | Airbyte has a PostgreSQL connector that can run incremental pulls every night and dump them as Parquet files partitioned by event_date. DMS gives true CDC but is more expensive. | All objects stored in an af‑south‑1 bucket – meets POPIA “data residency” requirement. |
| Metadata catalog | AWS Glue Data Catalog (linked to S3) | Lets Snowflake read the raw files with a source table (raw.<schema>.<table>). | ✅ |
| Warehouse | Snowflake on af‑south‑1 | • SQL‑first, ACID transactions. <br>• Native clustering / partitioning → very fast aggregations for LTV and cohort work. <br>• Built‑in Snowpark for any needed PySpark transformations. | All compute & storage stay inside the region. |
| Transformations | dbt (Snowflake adapter) | Declarative data model, lineage, tests, documentation. <br>Runs nightly jobs from dbt Cloud or GitHub Actions. | ✅ |
| Feature store / ML ingestion | Feast + Snowflake (or SageMaker Feature Store) | For churn‑prediction you need a snapshot of customer features that can be served to your model in real time. <br>Feast integrates with Snowflake, giving a single source for the training data and online feature store. | ✅ |
| Analytics & BI | Looker Studio (free) → Snowflake OR Power BI + Snowflake connector | Executive dashboards (LTV, churn trend, cohort curves). <br>Looker Studio is low‑cost and integrates out of the box with Snowflake. | ✅ |
| Model training / ops | SageMaker or Vertex AI (if you prefer GCP) – or custom Jupyter on EC2/ECS | Trains churn models using the snapshot from Feast. <br>Training data lives in Snowflake → exported as Parquet to S3; SageMaker pulls it via a role. | If you want everything inside AWS, use SageMaker. |
Data flow diagram (text)
``
┌───────────────────────┐
│ PostgreSQL (source) │
└─────────▲─────────────┘
│
Airbyte CDC / Incremental Pull
▼
┌───────────────────────┐ ───► Snowflake raw schema
│ S3 bucket (Parquet) │
└─────────▲─────────────┘
│
Glue Data Catalog
▼
┌───────────────────────┐ ───► Snowflake staging & marts
│ dbt projects │
└─────────▲─────────────┘
│
Snowflake warehouse (Warehouse + Feature Store)
▼
┌───────────────────────┬───────────────────────┐
│ BI / Dashboards │ ML training │
│ Looker Studio / Power│ SageMaker / Feast │
└───────────────────────┴───────────────────────┘
`
---
2️⃣ Dimensional Model
Below is a star schema that covers LTV, churn and cohort work.
All dimension tables use surrogate keys (*_SK); the fact tables reference them.
| Table | Granularity | Key fields | Notes |
|-------|-------------|------------|-------|
| dim_customer | 1 row per customer | customer_sk, customer_id, first_name, last_name, email, created_at, status (ACTIVE / INACTIVE), segment, signup_channel | Keeps a stable lookup for all facts. |
| dim_product | 1 row per product SKU | product_sk, sku, name, category, price, brand | Add if you want spend by product or category. |
| dim_time | 1 day row | time_sk, date, day_of_week, week, month, quarter, year | Drives all time‑series aggregations. |
| dim_region | 1 row per region | region_sk, country, state, city | Useful for geography‑driven churn signals. |
| dim_channel | 1 row per acquisition channel | channel_sk, channel_name, source_type | E.g. FACEBOOK, ORGANIC, EMAIL. |
Fact Tables
| Table | Purpose | Key fields (FK) | Measures | Notes |
|-------|---------|-----------------|----------|-------|
| fact_sales | Individual transaction (used for LTV & churn metrics) | sale_sk, time_sk, customer_sk, product_sk, region_sk | quantity, unit_price, discount_amount, net_revenue | Partition by time_sk; cluster on customer_sk. |
| fact_customer_activity | Non‑transactional events (logins, support tickets) | activity_sk, time_sk, customer_sk | activity_type enum (LOGIN, SUPPORT_TICKET, WEB_SESSION), count | Enables churn features like recency and engagement. |
| fact_ltv | Pre‑aggregated LTV per customer & time slice (monthly) | ltv_sk, customer_sk, time_sk | cumulative_revenue | Incremental dbt model; refreshed nightly. |
Derived Fact / Snapshot for Churn Features
| Table | Purpose | Key fields |
|-------|---------|------------|
| snapshot_customer_features | Daily snapshot of RFM, last‑active date, support tickets count etc. | customer_sk, snapshot_date | Materialised as a dbt incremental model; feeds Feast or your ML training data store. |
> Why both fact_sales & fact_ltv?
> fact_sales keeps the atomic level for audit & detailed queries (e.g., “show me all purchases by customer X in June”).
> fact_ltv gives a ready‑to‑use view that is far cheaper to query when you just need the cumulative value per month.
---
3️⃣ Sample DDLs (Snowflake syntax)
`sql
-- 1. Dimensions -------------------------------------------------
CREATE TABLE dim_customer (
customer_sk NUMBER AUTOINCREMENT,
customer_id VARCHAR(50) NOT NULL, -- natural key
first_name STRING,
last_name STRING,
email STRING,
created_at TIMESTAMP_LTZ,
status STRING, -- ACTIVE / INACTIVE
segment STRING,
signup_channel STRING,
PRIMARY KEY (customer_sk),
UNIQUE (customer_id)
);
CREATE TABLE dim_product (
product_sk NUMBER AUTOINCREMENT,
sku VARCHAR(50) NOT NULL,
name STRING,
category STRING,
price FLOAT,
brand STRING,
PRIMARY KEY (product_sk),
UNIQUE (sku)
);
CREATE TABLE dim_time (
time_sk NUMBER AUTOINCREMENT,
date DATE NOT NULL,
day_of_week VARCHAR(10),
week INTEGER,
month INTEGER,
quarter INTEGER,
year INTEGER,
PRIMARY KEY (time_sk),
UNIQUE (date)
);
-- 2. Facts -------------------------------------------------------
CREATE TABLE fact_sales (
sale_sk NUMBER AUTOINCREMENT,
time_sk NUMBER NOT NULL, -- FK to dim_time
customer_sk NUMBER NOT NULL, -- FK to dim_customer
product_sk NUMBER NOT NULL, -- FK to dim_product
region_sk NUMBER, -- optional FK to dim_region
quantity INTEGER,
unit_price FLOAT,
discount_amount FLOAT,
net_revenue FLOAT,
PRIMARY KEY (sale_sk),
INDEX idx_sales_date (time_sk)
);
-- cluster on customer for LTV scans
ALTER TABLE fact_sales CLUSTER BY (customer_sk);
CREATE TABLE fact_customer_activity (
activity_sk NUMBER AUTOINCREMENT,
time_sk NUMBER NOT NULL,
customer_sk NUMBER NOT NULL,
activity_type STRING, -- LOGIN / SUPPORT_TICKET ...
count INTEGER DEFAULT 1,
PRIMARY KEY (activity_sk)
);
CREATE TABLE fact_ltv (
ltv_sk NUMBER AUTOINCREMENT,
time_sk NUMBER NOT NULL,
customer_sk NUMBER NOT NULL,
cumulative_revenue FLOAT,
PRIMARY KEY (ltv_sk)
);
`
> Tip: In Snowflake you can create view vw_customer_ltv_monthly that joins fact_ltv, dim_customer, dim_time. This keeps your dashboards lean.
---
4️⃣ dbt Transformation Flow
- Staging – raw tables from S3 → snowflake staging schemas (stg_...
).
E.g. stg_transactions (raw Parquet from Airbyte).
- Intermediate – data cleaning, type casting, de‑duplication.
E.g. int_sales_cleaned.
- Marts / Dimensional – the final dimensional tables (dim_customer
, dim_product, …).
Model:
`sql
-- models/marts/core/dim_customer.sql
{{ config(materialized='table') }}
SELECT
ROW_NUMBER() OVER (ORDER BY customer_id) AS customer_sk,
customer_id,
first_name,
last_name,
email,
created_at,
status,
segment,
signup_channel
FROM {{ ref('stg_customers') }}
`
- Fact – merge dimensions + facts; use incremental models for daily loads.
`sql
-- models/facts/fact_sales.sql
{{ config(materialized='incremental', unique_key='sale_id') }}
SELECT
ROW_NUMBER() OVER (ORDER BY sale_id) AS sale_sk,
DATE_TRUNC('day', transaction_ts)::DATE as txn_day, -- will join to dim_time
customer_sk,
product_sk,
region_sk,
quantity,
unit_price,
discount_amount,
net_revenue
FROM {{ ref('int_sales_cleaned') }} s
JOIN {{ ref('dim_customer') }} c USING (customer_id)
JOIN {{ ref('dim_product') }} p USING (sku)
LEFT JOIN {{ ref('dim_region') }} r USING (region_code)
WHERE NOT EXISTS (
SELECT 1 FROM fact_sales fs WHERE fs.sale_id = s.sale_id
)
`
- Derived snapshots – churn features:
`sql
-- models/snapshots/customer_features.sql
{{ config(materialized='incremental', unique_key=['customer_sk','snapshot_date']) }}
WITH recency AS (
SELECT customer_sk,
MAX(transaction_ts) AS last_purchase_ts
FROM {{ ref('fact_sales') }}
GROUP BY customer_sk
),
frequency AS (
SELECT customer_sk,
COUNT(DISTINCT DATE_TRUNC('day', transaction_ts)) AS purchase_days
FROM {{ ref('fact_sales') }}
GROUP BY customer_sk
)
SELECT c.customer_sk,
CURRENT_DATE() AS snapshot_date,
COALESCE(r.last_purchase_ts, '1970-01-01')::DATE as last_purchase_date,
f.purchase_days,
(SELECT COUNT(*) FROM {{ ref('fact_customer_activity') }} a
WHERE a.customer_sk = c.customer_sk AND a.activity_type='LOGIN')
AS daily_logins_last_30d
FROM {{ ref('dim_customer') }} c
LEFT JOIN recency r USING (customer_sk)
LEFT JOIN frequency f USING (customer_sk);
`
- Testing – dbt tests (not_null
, unique, custom test for monotonic LTV growth).
`yaml
models/facts/schema.yml
models:
columns:
tests:
`
---
5️⃣ ML / Churn Pipeline
- Feature Store – Feast
- Pull the snapshot table (snapshot_customer_features
) into Feast’s online store.
- Each feature is stored with its current value + timestamp → real‑time scoring possible.
- Training Data Export – Every week, export the last 30 days of features and target label (churned in next 90 days). Load to S3 as Parquet.
- Model Training – SageMaker / Vertex AI
- Use XGBoost or LightGBM.
- Log hyper‑parameters and metrics with MLflow (integrates natively with SageMaker).
- Model Registry & Deployment – SageMaker Model Registry; deploy as a real‑time endpoint (mlops_endpoint
).
- Drift Monitoring – Feast’s batch feature store can compute PSI for each feature every day. Alert when > 0.25 → retrain.
---
6️⃣ Governance & Security
| Area | Tool / Practice |
|------|-----------------|
| Data residency | All S3, Snowflake, Glue in af-south‑1. |
| Access control | Snowflake roles: SYSADMIN, DATAENGINEER, BI_READER, ML_OPERATOR. Least privilege on every table. |
| Audit logs | CloudTrail + Snowflake query history → detect unauthorized reads/writes. |
| Data quality | dbt tests, Great Expectations for raw lake. |
| Metadata catalog | AWS Glue (linked to Snowflake) + dbt docs. |
| PII handling | Store PII only in dim_customer; encrypt all columns with dynamic data masking; anonymise logs before moving to the feature store. |
---
7️⃣ Quick Implementation Checklist
- Set up S3 bucket (af‑south‑1) – enable versioning, SSE‑S3.
- Create Airbyte / DMS job → pull PostgreSQL nightly into s3://my-bucket/raw/<date>/
.
- Create Glue tables for each Parquet dataset.
- Provision Snowflake account in the same region; create warehouse (small/medium as per load).
- Load raw data from S3 to Snowflake staging schema using COPY INTO
– schedule nightly.
- Spin up dbt Cloud or GitHub Actions – connect to Snowflake, run nightly DAGs: dbt run
, dbt test.
- Build dashboards in Looker Studio using the dim_
and fact_ tables; publish to executive users.
- Set up Feast – point to Snowflake as data source for offline store; configure online store (Redis).
- Train a baseline churn model in SageMaker → register → deploy endpoint.
- Configure alerts: PSI > 0.25, LTV decline > 5 % month‑over‑month.
---
8️⃣ Bottom‑Line Takeaway
- Star schema with dim_customer
, dim_product, dim_time, dim_region + three fact tables (fact_sales, fact_customer_activity, fact_ltv) gives you everything you need for LTV, churn & cohort analysis.
- Batch ELT (PostgreSQL → S3 Parquet → Snowflake staging) is simple, reliable and fits 50 M rows.
- dbt keeps transformations reproducible; use tests to guard against drift before it hits production dashboards or the model training data.
- Feast + SageMaker gives a clean ML‑ops path for churn prediction without needing custom infra.
- All components live in South African AWS (af-south‑1`), satisfying POPIA and latency requirements.
Feel free to tweak sizes (warehouse size, clustering columns) once you start seeing actual query performance numbers. The above design is the minimum viable architecture that scales cleanly as your data volume grows beyond 50 M rows.