Data Pipeline Platform
Kafka-based event pipeline built in explicit tiers. Producer, consumer, and metrics are working; the reliability tiers are not built yet.
A Kafka pipeline built in explicit tiers so the page can say exactly which reliability work is done.
- Role
- Solo: building
- Timeframe
- Jan 2026 to present
- repo created 2026-01-26
- Team
- 1
- Problem
- Event pipelines are easy to describe as handling millions of events a day and hard to actually make reliable.
- Approach
- Four tiers: tracer bullet first, then dead letter queue, schema evolution and backpressure, then multi-source ingestion, then reliability drills and SLOs.
- Result
- Tier 0 works; tier 1 is partly built (producer, consumer, metrics). No throughput numbers until tier 3 produces them.
Technology Stack
Where it actually is
The project is structured as four tiers, and I am partway through the second. T0 (local stack plus an end-to-end tracer-bullet flow) works. T1 adds a dead letter queue, schema evolution, backpressure handling, and basic observability. The producer and consumer with metrics are committed, the rest is in progress.
T2 (multi-source ingestion, backfills, hot/cold storage, orchestration) and T3 (reliability drills, SLOs, scale tests, a capacity model) are planned and not started.
Why the tiers are public
It would be easy to describe this as a system 'handling millions of events per day'. It is not doing that, and I would rather the page match the repository.
The tier structure is the honest version and I think it is also the more useful one: it states what reliability work is done, what is next, and what 'finished' means. Throughput numbers go on this page when T3 scale tests produce them.