Adtech data platform

The hard problem lived in the tail.

At the order of billions of events per day, a small number of high-volume partitions could dominate a Spark stage and make an otherwise healthy batch unpredictable. The platform also had to keep downstream APIs useful while the data moved through the system.

The figures here are deliberately order-of-magnitude: exact throughput and latency details are rounded for confidentiality.

I made the slow path legible.

I worked across the Spark ETL and API layers I owned, profiling stage behavior, shaping the data path around its busiest partitions, and adding observability so a slow tail became evidence an engineer could act on rather than a vague batch alarm.

Batch the heavy work; keep APIs thin.

I kept the high-volume transformation in Spark instead of moving it into per-request API work. The rejected alternative would have made the batch simpler on paper, but it would have paid for the same computation repeatedly and pushed unpredictable work into the user-facing path.

A healthy average hid an unhealthy tail.

The aggregate pipeline looked fine until one skewed path stretched the batch. Stage-level instrumentation and tests around that path gave the team a way to see the outlier early and reason about it without guessing.

Standardize the contract sooner.

Next time I would cut bespoke telemetry at the service edges and establish one shared freshness and failure contract earlier. The platform had more surface area than every consumer needed; a smaller contract would have made the system easier to operate.