Projects
Public, reproducible work. Each repo has a design doc, tests and CI.
Featured · Data engineering + AI
A governed lakehouse for semiconductor manufacturing test data (UCI SECOM, 1,567 wafer runs × 590 sensors). It goes from raw sensor readings to one trusted yield metric, a ranked list of suspect sensors, multivariate anomaly scores, and plain-English questions answered by guardrailed, benchmarked natural-language-to-SQL.
- Bronze → silver → gold with an explicit grain at every layer: Kimball dimensions, wafer and sensor facts, yield defined exactly once
- Business-rule tests catch outputs that are "technically valid but semantically wrong": yield ∈ [0,1], fails ≤ wafers, every wafer has readings
- The same dbt models run on DuckDB (local and CI) and Databricks + Delta Lake (cloud)
- The NL-to-SQL guardrail allows only single SELECTs against approved gold models; execution accuracy is scored against gold SQL
- dbt Core
- Databricks
- Delta Lake
- DuckDB
- Python
- scikit-learn
- GitHub Actions
Featured · Analytics engineering + finance
A public rebuild, on synthetic data, of the accounting analytics I did for a food-delivery marketplace. An incremental dbt pipeline ties a multi-currency commission sub-ledger (91k events) to the general ledger with $0 unexplained variance on all 26 account-months, explains every difference as a reconciling item, and proposes reclassification journals for revenue booked to the wrong account.
- An answer-key test plants a duplicate journal, a wrong-FX posting, a late month-end posting and an account-mapping bug. The build fails unless every one is found with the right reason and amount, and nothing else is flagged.
- The revenue fact is incremental on load time, not event time. A script proves the incremental result equals a full refresh, including events that arrive late.
- FX is translated at batch level, as the GL does it, with a 5-day fallback for weekends and holidays; dbt unit tests cover the fallback and the variance rules.
- Dagster assets with a daily schedule and a month-end re-tie job; the same SQL targets DuckDB (local and CI) and Snowflake.
- dbt Core
- DuckDB
- Snowflake
- Dagster
- Python
- GitHub Actions
Featured · Analytics engineering + fintech
Six continuous-audit controls built as dbt models and measured on 6.36M mobile-money transactions (PaySim, 8,213 fraud). Two controls kept for production catch 99.4% of fraud at 89.8% precision. The simple over-200k transfer rule would raise 409k alerts at 0.7% precision. On a later, unseen time window, a gradient-boosted model only matches the two rules, so the explainable rules stay.
- Every control is scored for precision and recall against the fraud label. Each gets a production, review or retired status from those results, with the reason recorded. A balance-integrity check that raised 1.18M alerts for 20 frauds was retired from the fraud queue.
- An exceptions queue with severity, evidence and review status, plus a dbt snapshot that keeps the SCD2 history of every review decision as an audit trail.
- Answer-key tests on planted mule chains, duplicate payments and fan-in accounts; unit-tested control logic; an enforced contract on the performance report. The full build runs in about 10 seconds on DuckDB.
- Benford's-law conformity by transaction type, graded with Nigrini's thresholds, plus a frank note on what simulated data can and can't prove.
- dbt Core
- DuckDB
- Python
- scikit-learn
- GitHub Actions
Featured · Analytics engineering + finance
A public rebuild, on synthetic data, of the accounts-receivable analytics I moved from Tableau to Streamlit on dbt at a food-delivery marketplace. A dbt model of 660k invoices across 25,000 customers gives aging, DSO, CEI, unapplied cash and a ranked collections worklist. Every screen of the app proves the sub-ledger ties to the general ledger (54 of 54 month-end and currency checks, to the cent).
- Month-end open items for every invoice. Aging is reported by the segment each customer had at that month-end, through a point-in-time join to a type-2 customer dimension.
- The cash-application fact is incremental on load time. A script proves the incremental run equals a full rebuild across 650k rows, including payments that load days late.
- 59 dbt checks run in about 4 seconds, covering the GL tie-out, cash never over-applied, aging totals and KPI ranges, plus unit tests on the type-2 windows and the month-end balance logic.
- A Streamlit app with KPI trends, aging, a collections worklist with customer drill-down, and a pipeline-health tab built from recorded dbt runs.
- dbt Core
- DuckDB
- Streamlit
- Python
- GitHub Actions
Featured · Analytics engineering + controls
A public rebuild, on synthetic data, of the risk-rating control app I built to replace a spreadsheet upload for ~20k merchant accounts. Ratings for 20,000 merchants come from scoring rules held as data, and overrides go through a maker-checker approval app with required reasons and period locks. Every action lands in an append-only audit log that dbt re-audits independently: 0 control violations across 2,994 events.
- Eight scoring rules live in a dbt seed. Signals are unpivoted and joined to the rules, so thresholds and weights change without SQL, and every rating lists the rules behind it.
- Controls are enforced twice: in the app (20 pytest tests: self-approval, roles, reasons, locks, duplicates) and again in SQL over the whole log. An answer-key test plants five kinds of breach and requires each to be caught.
- A reconciliation guard on "newly high-risk" counts: a silently failed prior-month join would overstate them 1.9–2.4× on this data, and the build now fails if that happens.
- Type-2 rating history (gaps and islands) and a dbt snapshot. I verified the full flow in the running app: propose, refused self-approval, checker approval, rating changed.
- dbt Core
- DuckDB
- Streamlit
- pytest
- Python
- GitHub Actions
Featured · Data engineering + IoT reliability
The fleet-reliability approach I led for ~680k devices, applied to real data: 30.6 million drive-days from Backblaze's 351,000-drive fleet (Q1 2026). It produces annualised failure rates with confidence intervals, failures per 1,000 units, the age "bathtub" curve, left-truncated survival curves, and a next-quarter replacement forecast (914 expected failures, against 1,027 observed this quarter).
- The 30.6M rows load in 23 seconds with DuckDB. dbt builds a drive-day fact and AFR by model, age and manufacturer: 1.24% fleet AFR, with 35 models meeting Backblaze's reporting threshold.
- Conformed keys: 476 drives report a WDC model without its prefix, which would split one model in two. Silver fixes it, keeps the raw value, and a test guards it.
- Governance built in: owners and classifications as dbt meta, enforced contracts on the AFR reports, a source profile, lineage, and source-quality warnings.
- SMART 197 and 198 appear on failed drives about 28× more often than on healthy ones. Kaplan–Meier is corrected for drives already in service (left truncation), and the README is open about where single-quarter curves are noisy.
- DuckDB
- dbt Core
- Python
- lifelines
- GitHub Actions
Featured · Data engineering + streaming
The streaming version of that device reporting: alerts as devices fail, not in the next day's report. 30.7 million real heartbeats (the Backblaze fleet, replayed with deliberate jitter, late arrivals and duplicate deliveries) go through event-time windows with a watermark, deduplication and a late-data side output, at 148k events a second. Every one of 540 windows reconciles to a dbt batch reference, and the batch matches the source's 1,030 failures exactly.
- Runs end to end through real Kafka (Redpanda) in CI: JSON schema in the registry, a 3-partition topic, offsets committed only after the sink write, then the same dbt reconciliation. Median latency from producer to published window is 0.41 s.
- The Kafka run found a real bug the ordered replay couldn't: 59% of events went late because the watermark followed the fastest partition. Per-partition watermarks (as in Flink) cut that to 0.3%.
- Effectively-once results: at-least-once input, deduplication and an idempotent upsert keyed by site and window. An Iceberg table keeps the history of every published window, and 10 pytest tests cover the engine.
- A mutation test proves the reconciliation catches lost events. A design doc covers batch vs streaming, lateness vs latency, and what changes at 10×.
- Kafka
- Redpanda
- Iceberg
- DuckDB
- dbt Core
- Python
- Streamlit
Featured · Analytics engineering + product
Product analytics on 2.76 million real e-commerce events (RetailRocket), with sessions, funnel, day-N retention and stickiness each defined once in dbt. Alongside it, an A/B testing toolkit validated on 269,683 real visitors: A/A tests, planted effects and a deliberately broken split. A product write-up argues which metric to move and why.
- Funnel: 2.5% of viewing sessions add to cart, 32.6% of those buy, and 87% of visitors never return. Add-to-cart is the step to move, and its tests need about 3× fewer users than purchase-conversion tests.
- The toolkit (sample size, sample-ratio check, CUPED, guardrails) recovers planted lifts within 0.3 points with 93–97% interval coverage. CUPED raises power at a +5% lift from 85% to 96% where the pre-period predicts the outcome.
- Honest findings: CUPED does nothing for conversion (pre-period correlation 0.06), and a 2% guardrail can't be verified at this traffic. A 2% loss of treatment units is caught (p = 2e-10).
- 27 dbt checks run in 8.5 seconds, including unit tests on the 30-minute session rule and bot exclusion, and a test that every human event lands in exactly one session.
- dbt Core
- DuckDB
- Python
- SciPy
- pytest
- GitHub Actions
Featured · Analytics engineering + procurement
A spend cube on $7.48B of real federal contract obligations (41,980 USAspending actions, Department of the Interior FY2025). Each client extract maps to one standard spend line through a small dbt adapter. USAspending, SAP-style and Oracle-style extracts all run in a CI matrix, and every dollar reconciles back to source in a test.
- 497 of 8,144 suppliers (6.1%) hold 80% of spend. The 7,647-supplier tail holds $1.36B, and facilities & maintenance is the most fragmented category (2,531 suppliers).
- Supplier name matching scored against registry IDs: fuzzy matching lifts recall from 0.77 to 0.82 (best F1 0.71). Shared names cap precision near 0.63, so suppliers are keyed on the ID and roll up 10,373 entities into 10,180 parents.
- Classifying spend from descriptions alone, on awards held out of training: a TF-IDF model reaches 0.745 accuracy against PSC labels vs 0.226 for keyword rules.
- Competition (11.9% not competed), small-business share (52.9%) and working capital (days payable, late interest, early-payment float, on labelled synthetic payment dates). 31 dbt checks run in about 2 seconds.
- dbt Core
- DuckDB
- Python
- rapidfuzz
- scikit-learn
- GitHub Actions
Featured · Analytics engineering + AI
14 metrics defined once in a dbt semantic layer (MetricFlow) over $6.57B of real federal contract spend (41,980 USAspending actions). An MCP server gives an AI agent those metrics and no SQL, and CI proves dashboards match the metrics to the cent.
- Benchmark of 54 questions with answers hand-written in SQL, separately from the semantic layer: 44/44 answered correctly through the governed layer, and 10/10 bad requests refused (undefined metrics, raw columns, SQL injection, bulk exports). Queries take 17 ms (p50).
- A dashboard check compares 258 cells against the semantic layer and catches a planted legacy dashboard that reports gross obligations as "spend" ($408.7M vs $365.7M governed net for one bureau).
- A Claude agent loop and an MCP server share one tool definition. Guardrail refusals come back as readable errors so the agent can correct itself, tested end to end with scripted replies. Running the agent on the benchmark is the next step.
- dbt Core
- MetricFlow
- MCP
- Claude API
- DuckDB
- pytest
Featured · Data engineering + healthcare
A Kimball claims warehouse on 857,285 CMS DE-SynPUF claim lines ($863M paid) for 116,352 synthetic Medicare members. It has SCD2 member history, a pre-authorisation accumulating snapshot, PMPM cost, and identifiers that are hashed and policy-tested. The same dbt project runs on DuckDB and Postgres under a daily Airflow 3 DAG.
- Member history is built as SCD2 from yearly files: 279,711 versions, since 75% of members changed. Claims join the version valid on their service date, so 2008 inpatient PMPM shows $2.67 with no chronic conditions vs $759 with five or more.
- The pre-authorisation snapshot stops the clock while a request waits for information. Turnaround is checked against CMS's 72-hour / 7-day deadlines, and each request is matched to its admission the way a payer does. Synthetic events are generated on real admissions, and the controls recover the planted $32.3M paid without approval.
- Data-quality findings against CMS's own totals: 2010 inpatient claims are 5.0% below the summary file, and 278 dateless claim lines are quarantined, not dropped. A masking test fails the build if a raw id leaks, and pytest proves it does.
- CI runs three jobs: DuckDB, Postgres 16, and Airflow 3.2. The Airflow job runs the DAG twice for the same day and checks that row counts don't change.
- dbt Core
- Airflow 3
- Postgres
- DuckDB
- pytest
- GitHub Actions
Featured · Data engineering + ML platform
An end-to-end churn pipeline on the IBM Telco data (7,043 customers). Point-in-time features are built in dbt with leakage tests, served through Feast offline and online stores to models tracked and registered in MLflow, with batch scoring and Evidently drift reports.
- Training and serving are proven identical: Feast's point-in-time join reproduces the dbt features on all 41,341 training rows, and the online store matches the offline rows for all 5,163 customers scored (0 mismatches each).
- Leakage is measured, not just avoided: swapping two point-in-time columns for their extract-time values inflates PR AUC from 0.309 to 0.515 on an out-of-time test. A dbt test recomputes features from earlier months only and must match.
- Out-of-time results: logistic regression on real-derived features reaches 0.842 ROC AUC. The registered champion reaches 0.913 with 6.4x top-decile lift, using monthly usage that is synthetic, which the README says plainly.
- The drift report flags 8 of 26 features as the scoring population becomes survivors. A control split shows 0 of 26, so the alert isn't firing on noise.
- dbt Core
- Feast
- MLflow
- Evidently
- scikit-learn
- DuckDB
Featured · Data engineering + platform
Every release ONS has published of monthly GDP since October 2020 (69 vintages), plus CPIH and Bank of England rates, on Apache Iceberg. Each vintage is written to a branch, audited, then published to main and tagged for time travel. The AWS deployment is defined in Terraform.
- Write-audit-publish caught real upstream problems and kept them from readers: a GDP release missing total GDP (v14), one with conflicting duplicate values (v67), and a listed release with no file. Garbled rows and CPIH's zero placeholders are handled and counted.
- Revisions are measured on growth, so re-referencing the index doesn't count. First estimates of monthly GDP growth moved 0.21 pp on average and were revised up by +0.10 pp, mostly in the 2020–21 recovery. 10 of 71 months changed direction. CPIH was never revised, which the same model proves.
- Schema evolution: a new source widened the table while old snapshots stayed readable. Pulls are incremental and idempotent: 69 releases run in 14.5 s, and a rerun publishes nothing.
- Terraform covers S3, the Glue catalogue, Athena with a scan cap, ECS Fargate on an EventBridge schedule, SNS alerts and least-privilege IAM. CI validates it on every push, and a weekly canary pulls the live APIs.
- Apache Iceberg
- Terraform
- AWS
- dbt Core
- DuckDB
- GitHub Actions
More featured projects
Each project rebuilds work from my roles on public or synthetic data. Every repo is public and CI-tested; each card says what is built and what comes next.
Featured · Data engineering + energy
Goal: Half-hourly smart-meter data processed at scale to find likely energy theft: the gap between energy consumed and energy bought, per meter and feeder, ranked for inspection with revenue at risk.
- Built so far: a CI-tested pipeline on a synthetic test dataset (22/22 dbt checks passing). Consumed vs purchased energy per meter per week, ranked within feeder, and unbilled share per feeder.
- Next: Ingest the London half-hourly readings into Delta, partitioned by date and clustered by meter.
- PySpark
- Databricks
- Delta Lake
- dbt Core
- DuckDB
Featured · Analytics engineering + geospatial
Goal: Feeder assets on a map, customers matched to the network, and the standard reliability indices (SAIDI and SAIFI) by feeder and month, with a hex map of the worst-served areas.
- Built so far: a CI-tested pipeline on a synthetic test dataset (15/15 dbt checks passing). SAIDI and SAIFI by month and by feeder, with a check that outages never exceed customers served.
- Next: Load OSM substations and lines for one city into PostGIS.
- PostGIS
- GeoPandas
- H3
- dbt Core
- DuckDB
Featured · Data engineering + energy
Goal: Half-hourly national demand loaded incrementally with a 5-year backfill, a day-ahead forecast benchmarked against a naive baseline, forecast accuracy tracked over time, and a morning ops report with no manual steps.
- Built so far: a CI-tested pipeline on a synthetic test dataset (13/13 dbt checks passing). Daily peak and average demand, a completeness test, and a same-day-last-week baseline with MAPE by month.
- Next: Incremental daily loads from NESO; backfill five years.
- Airflow
- Dagster
- dbt Core
- LightGBM
- DuckDB
Featured · Analytics engineering + HR
Goal: Learning and development analytics: training cost per employee, a competency-gap matrix and attrition cohorts by training history.
- Built so far: a CI-tested pipeline on a synthetic test dataset (16/16 dbt checks passing). Training cost per employee by year and department, with a trained-before-hired check.
- Next: Load the IBM attrition data and generate training history.
- dbt Core
- DuckDB
- Evidence.dev