Uber's Failover Architecture: Reconciling Reliability and Efficiency in Hyperscale Microservice Infrastructure

TL;DR

Uber's Failover Architecture (UFA) innovatively balances reliability and efficiency by differentiating service criticality, reducing over-provisioning from 2x to 1.3x, and maintaining high availability in hyperscale microservice infrastructure.

Contribution

UFA introduces a differentiated failover approach that optimizes resource utilization while ensuring critical service reliability at Uber.

Findings

Reduced steady-state provisioning from 2x to 1.3x

Increased resource utilization from ~20% to ~30%

Maintained 99.97% availability despite reduced capacity

Abstract

Operating a global, real-time platform at Uber's scale requires infrastructure that is both resilient and cost-efficient. Historically, reliability was ensured through a costly 2x capacity model--each service provisioned to handle global traffic independently across two regions--leaving half the fleet idle. We present Uber's Failover Architecture (UFA), which replaces the uniform 2x model with a differentiated architecture aligned to business criticality. Critical services retain failover guarantees, while non-critical services opportunistically use failover buffer capacity reserved for critical services during steady state. During rare "full-peak" failovers, non-critical services are selectively preempted and rapidly restored, with differentiated Service-Level Agreements (SLAs) using on-demand capacity. Automated safeguards, including dependency analysis and regression gates, ensure critical services continue to function even while non-critical services are unavailable. The quantitative impact is significant: UFA reduces steady-state provisioning from 2x to 1.3x, raising utilization from ~20% to ~30% while sustaining 99.97% availability. To date, UFA has hardened over 4,000 unsafe dependencies, eliminated over one million CPU cores from a baseline of about four million cores.