Cloudflare’s Code Orange: Fail Small Enhances Resilience and Incident Response

Cloudflare’s internal Code Orange: Fail Small initiative, completed in early 2026, delivers a suite of resilience improvements that prevented the November 18 and December 5 2025 global outages. The program introduced Snapstone, a health‑mediated deployment system that bundles configuration changes into packages and rolls them out progressively with real‑time health monitoring, replacing instant network‑wide configuration changes. Snapstone allows teams to define any configuration unit—data files, control flags, or other settings—on demand and ensures safe deployment across the network. In addition, Cloudflare now implements fail‑open and fail‑close logic for critical services, using a fail‑stale fallback to the last known good configuration when possible.

The initiative also segments the Workers runtime into independent services per customer cohort, allowing changes to roll out first to free customers and then to more critical segments, thereby limiting blast radius. Backup authorization pathways for 18 key services were added, along with emergency scripts and proxies, to maintain visibility and control during outages. An engineering‑wide drill on April 7, 2026, tested these break‑glass procedures, and a dedicated communications team was established to streamline customer updates during incidents. Finally, Cloudflare codified all lessons learned into an internal Codex to prevent drift and regressions.

These enhancements provide safer configuration deployments, graceful failure handling, and faster incident resolution, directly benefiting all customers by reducing downtime and improving reliability.