Rewinding the exponential: How we keep hardware usage down
Our strategies to optimize our data pipeline throughput without compromising data quality.
Christian Endres
Our billing pipeline was suddenly slow. The culprit was a hidden bottleneck in ClickHouse
When a partitioning change to our petabyte-scale ClickHouse cluster caused critical billing jobs to stall, standard metrics showed no obvious errors. This post explores how we identified severe lock contention in ClickHouse's query planner and built upstream patches to fix it.
Cloudflare incident on November 14, 2024, resulting in lost logs
On November 14, 2024, Cloudflare experienced a Cloudflare Logs outage, impacting the majority of customers using these products. During the ~3.5 hours that these services were impacted, about 55% of the logs we normally send to customers were not sent and were lost. The details of what went wrong and why are interesting both for customers and practitioners.
