PolarStore: High-Performance Data Compression for Large-Scale Cloud-Native Databases

TL;DR

PolarStore introduces a dual-layer compression system combining hardware and software techniques to significantly reduce storage costs in cloud-native RDBMSs without sacrificing performance.

Contribution

It presents PolarStore, a novel hybrid compression system with database optimizations and hardware improvements, enabling high compression ratios and cost savings in large-scale cloud databases.

Findings

Achieves a 3.55 compression ratio.

Reduces storage costs by approximately 60%.

Maintains high performance comparable to uncompressed systems.

Abstract

In recent years, resource elasticity and cost optimization have become essential for RDBMSs. While cloud-native RDBMSs provide elastic computing resources via disaggregated computing and storage, storage costs remain a critical user concern. Consequently, data compression emerges as an effective strategy to reduce storage costs. However, existing compression approaches in RDBMSs present a stark trade-off: software-based approaches incur significant performance overheads, while hardware-based alternatives lack the flexibility required for diverse database workloads. In this paper, we present PolarStore, a compressed shared storage system for cloud-native RDBMSs. PolarStore employs a dual-layer compression mechanism that combines in-storage compression in PolarCSD hardware with lightweight compression in software. This design leverages the strengths of both approaches. PolarStore also incorporates database-oriented optimizations to maintain high performance on critical I/O paths. Drawing from large-scale deployment experiences, we also introduce hardware improvements for PolarCSD to ensure host-level stability and propose a compression-aware scheduling scheme to improve cluster-level space efficiency. PolarStore is currently deployed on thousands of storage servers within PolarDB, managing over 100 PB of data. It achieves a compression ratio of 3.55 and reduces storage costs by approximately 60%. Remarkably, these savings are achieved while maintaining performance comparable to uncompressed clusters.