Pipewright: End-to-End Data Pipeline for a CEO Dashboard
BazaarBox, a direct-to-consumer home & kitchen brand, wants one dashboard that is correct by 06:30 IST every day: revenue by category and region, CAC by channel, Prime membership MRR, the top 10 products of the last 30 days and 30-day cohort retention. Over nine phases you build the pipeline behind it like a data engineering team: write requirements with SLAs and acceptance criteria, design the architecture, the bronze/silver/gold layers, a data contract and five ADRs (including why Airbyte, a Spark cluster and Metabase were swapped for lighter tools), then build it test-first. A seeded, offline copy of BazaarBox's systems (an orders and customers API with bearer tokens, cursor pagination, updated_at windows and injected 429/503 faults, plus daily payment, ad-spend and catalogue file drops) comes with real-world defects: mixed time zones, re-delivered orders, duplicate payment events, rupee amounts as text, city spellings, test orders and PII. You write idempotent ingestion into immutable lake partitions, silver and gold models in dbt (SQL track) or PySpark (Python track) whose gold tables must equal the golden files and each other row by row, declarative data-quality checks with severities and a publish gate built on an atomic swap, dbt data and unit tests, and a GitHub Actions pipeline that runs dbt build on a fixture and both tracks on the full history. v1.0.0 adds an Airflow 3 schedule, a Streamlit dashboard, an optional AI incident explainer with guardrails and an offline fallback, and a hardened Docker Compose stack under 6 GB of RAM. Then the pipeline meets its second month: a new channel and a new order status block publishing, coupons change the schema, and a partner file arrives late. You version the contract additively, make the facts incremental and prove them equal to a full refresh, add backfills, freshness, volume and SLA monitoring with alerts, prove backward compatibility with v1.0, and write a blameless post-mortem (v1.1.0). Everything runs on a laptop with no paid service and no API key.
What you will build, step by step
- Phase 0 · Setup, the brief and the sources
- Phase 1 · Requirements: the five reports, SLAs and acceptance criteria
- Phase 2 · Design: architecture, the data contract, layers and ADRs
- Phase 3 · Implementation in four milestones: ingest, silver, gold, run
- Phase 4 · Data tests and data-quality checks: the publish gate
- Phase 5 · CI: dbt build and tests on the fixture, cross-track parity
- Phase 6 · v1.0: the scheduled pipeline, the dashboard, the AI explainer, Docker
- Phase 7 · v1.1: incremental models, backfills, schema evolution, monitoring and alerts
- Phase 8 · Retrospective and portfolio