TierBase: A Workload-Driven Cost-Optimized Key-Value Store

TL;DR

This paper presents TierBase, a distributed key-value store that uses a workload-driven cost model and optimization techniques to significantly reduce storage costs while maintaining high performance in large-scale data environments.

Contribution

It introduces a novel cost model guiding storage configuration and develops TierBase, a system that optimizes resource utilization through data tiering and advanced techniques.

Findings

Achieves up to 62% cost reduction in production environments

Demonstrates superior cost-effectiveness over existing solutions

Effectively handles skewed workloads with optimized resource management

Abstract

In the current era of data-intensive applications, the demand for high-performance, cost-effective storage solutions is paramount. This paper introduces a Space-Performance Cost Model for key-value store, designed to guide cost-effective storage configuration decisions. The model quantifies the trade-offs between performance and storage costs, providing a framework for optimizing resource allocation in large-scale data serving environments. Guided by this cost model, we present TierBase, a distributed key-value store developed by Ant Group that optimizes total cost by strategically synchronizing data between cache and storage tiers, maximizing resource utilization and effectively handling skewed workloads. To enhance cost-efficiency, TierBase incorporates several optimization techniques, including pre-trained data compression, elastic threading mechanisms, and the utilization of persistent memory. We detail TierBase's architecture, key components, and the implementation of cost optimization strategies. Extensive evaluations using both synthetic benchmarks and real-world workloads demonstrate TierBase's superior cost-effectiveness compared to existing solutions. Furthermore, case studies from Ant Group's production environments showcase TierBase's ability to achieve up to 62% cost reduction in primary scenarios, highlighting its practical impact in large-scale online data serving.