Cloudflare Uncovers ClickHouse Lock Contention Bottleneck, Patches Performance
Patch the MergeTreeData mutex contention by changing the planner to use a read‑only snapshot of parts instead of copying the full list.
Patch the MergeTreeData lock contention by updating the planner code to avoid copying the full parts list.
Summary
Cloudflare relies on ClickHouse for petabyte‑scale analytics, with the Ready‑Analytics table holding over 2 PiB of data. To support per‑namespace retention, the team added the namespace field to the partition key, changing it from (day) to (namespace, day). The migration began in January 2025 using the Merge table feature, writing new data to the new partitioned table while old data aged out.
In March 2025 billing jobs slowed dramatically; investigation revealed a linear growth in total part count. Flame graphs showed 45 % of CPU time spent in filterPartsByPartition, and a subsequent real‑trace flame graph revealed that more than half of query duration was waiting to acquire the MergeTreeData mutex that protects the full parts list. The root cause was lock contention when every thread copied the entire parts list.
Three patches were written: a heuristic reorder that yielded a 5 % improvement, a lock‑contention fix that eliminated the mutex wait, and the new partitioning scheme. The max‑min fairness algorithm allowed the cluster to run at 90 % utilization. After the patches, the billing jobs returned to normal speed and the system could scale to petabyte‑level workloads without performance regressions.
Key changes
- Added namespace to the partition key, changing it to (namespace, day).
- Implemented per‑namespace retention by dropping partitions older than 31 days per namespace.
- Identified lock contention on the MergeTreeData mutex during query planning.
- Flame graphs showed 45% CPU time in filterPartsByPartition and >50% wait on the mutex.
- Reordered heuristics in the planner for a 5% performance improvement.
- Implemented a lock‑contention fix that eliminates the mutex wait during planning.
- Used a max‑min fairness algorithm to share disk space and run clusters at 90% utilization.
- Migrated data using the Merge table feature, writing new data to the new partitioned table while old data aged out.