Implementing Decentralized Per-Partition Automatic Failover in Azure Cosmos DB

TL;DR

This paper presents a decentralized, fine-grained automatic failover mechanism for Azure Cosmos DB that enhances data availability and minimizes recovery time during regional outages at massive scale.

Contribution

It introduces a novel decentralized architecture enabling partition-level geo failover with minimal RTO and RPO, handling diverse fault scenarios at scale.

Findings

Supports seamless failover across regions

Minimizes recovery time during outages

Handles various hardware and network faults

Abstract

Azure Cosmos DB is a cloud-native distributed database, operating at a massive scale, powering Microsoft Cloud. Think 10s of millions of database partitions (replica-sets), 100+ PBs of data under management, 20M+ vCores. Failovers are an integral part of distributed databases to provide data availability during outages (partial or full regional outages). While failovers within a replica-set within a single region are well understood and commonly exercised, geo failovers in databases across regions are not as common and usually left as a disaster recovery scenario. An upcoming release of Azure Cosmos DB introduces a fine grained (partition-level) automatic failover solution for geo failovers that minimizes the Recovery Time Objective (RTO) and honors customer-chosen consistency level and Recovery Point Objective (RPO) at any scale. This is achieved thanks to a decentralized architecture which offers seamless horizontal scaling to allow us to handle outages ranging from node-level faults to full-scale regional outages. Our solution is designed to handle a broad spectrum of hardware and software faults, including node failures, crashes, power events and most network partitions, that span beyond the scope of a single fault domain or an availability zone.