Equilibrium: Optimization of Ceph Cluster Storage by Size-Aware Shard Balancing

TL;DR

This paper introduces Equilibrium, a size-aware shard balancing algorithm for Ceph clusters that improves storage capacity utilization and reduces data movement through extensive experiments.

Contribution

The paper presents a novel size-aware shard balancing algorithm for Ceph, enhancing storage efficiency and minimizing data movement compared to existing methods.

Findings

Achieves near-optimal balance on real-world clusters

Significantly improves available storage capacity

Reduces data movement during balancing

Abstract

Worldwide, storage demands and costs are increasing. As a consequence of fault tolerance, storage device heterogenity, and data center specific constraints, optimal storage capacity utilization cannot be achieved with the integrated balancing algorithm of the distributed storage cluster system Ceph. This work presents Equilibrium, a device utilization size-aware shard balancing algorithm. With extensive experiments we demonstrate that our proposed algorithm balances near optimally on real-world clusters with strong available storage capacity improvements while reducing the amount of needed data movement.