Marlin: Efficient Coordination for Autoscaling Cloud DBMS (Extended Version)

TL;DR

Marlin introduces a cloud-native coordination mechanism for disaggregated cloud databases, eliminating external services, improving scalability, cost efficiency, and reconfiguration speed through integrated coordination and transactional guarantees.

Contribution

Marlin is the first to fully integrate coordination into cloud-native databases, enabling storage disaggregation and eliminating external coordination services.

Findings

Up to 4.4x cost efficiency improvement

Reconfiguration duration reduced by up to 4.9x

Effective coordination without external services

Abstract

Modern cloud databases are shifting from converged architectures to storage disaggregation, enabling independent scaling and billing of compute and storage. However, cloud databases still rely on external, converged coordination services (e.g., ZooKeeper) for their control planes. These services are effectively lightweight databases optimized for low-volume metadata. As the control plane scales in the cloud, this approach faces similar limitations as converged databases did before storage disaggregation: scalability bottlenecks, low cost efficiency, and increased operational burden. We propose to disaggregate the cluster coordination to achieve the same benefits that storage disaggregation brought to modern cloud DBMSs. We present Marlin, a cloud-native coordination mechanism that fully embraces storage disaggregation. Marlin eliminates the need for external coordination services by consolidating coordination functionality into the existing cloud-native database it manages. To achieve failover without an external coordination service, Marlin allows cross-node modifications on coordination states. To ensure data consistency, Marlin employs transactions to manage both coordination and application states and introduces MarlinCommit, an optimized commit protocol that ensures strong transactional guarantees even under cross-node modifications. Our evaluations demonstrate that Marlin improves cost efficiency by up to 4.4x and reduces reconfiguration duration by up to 4.9x compared to converged coordination solutions.