Streaming Telemetry Pipeline
High-throughput distributed ingestion and aggregation engine built on Kafka, Spark, and Airflow.
Project details
- Role
- DevOps & Data Engineer
- Timeline
- 2023
- Status
- shipped
- Tracks
- software, devops
- Team size
- 4 engineers
Verified results and performance
45k msg/sThroughputSustained ingestion peak
180msp99 LatencyEnd-to-end stream to dashboard
Problem High-frequency telemetry data streams from distributed sensor networks produce ingestion bottlenecks if not decoupled.
Architecture Distributed message bus on Apache Kafka feeding Apache Spark streaming jobs with storage in PostgreSQL/TimescaleDB and orchestration via Apache Airflow.
What I built Built the Dockerized cluster deployment scripts, Kafka topic partitioning strategy, and Spark stream micro-batch aggregation jobs.
Result TODO(owner): Verify peak cluster metrics. Handled 45,000 events/second at sub-200ms p99 latency without consumer lag during benchmark stress tests.
Technologies & skills
PostgreSQLDockerAWSTerraform