Aster: Enhancing LSM-structures for Scalable Graph Database

TL;DR

Aster introduces Poly-LSM, a novel graph-oriented storage engine with adaptive and skew-aware techniques, significantly improving scalability and performance for large-scale, evolving graph databases like Twitter-scale graphs.

Contribution

The paper presents Poly-LSM, a new graph-oriented LSM-tree design integrated into Aster, enhancing scalability and efficiency for large, dynamic graph workloads.

Findings

Aster outperforms baseline graph databases on large-scale graphs.

Up to 17x throughput improvement on billion-scale Twitter graph.

Poly-LSM effectively handles graph updates with optimized I/O and skew-aware encoding.

Abstract

There is a proliferation of applications requiring the management of large-scale, evolving graphs under workloads with intensive graph updates and lookups. Driven by this challenge, we introduce Poly-LSM, a high-performance key-value storage engine for graphs with the following novel techniques: (1) Poly-LSM is embedded with a new design of graph-oriented LSM-tree structure that features a hybrid storage model for concisely and effectively storing graph data. (2) Poly-LSM utilizes an adaptive mechanism to handle edge insertions and deletions on graphs with optimized I/O efficiency. (3) Poly-LSM exploits the skewness of graph data to encode the key-value entries. Building upon this foundation, we further implement Aster, a robust and versatile graph database that supports Gremlin query language facilitating various graph applications. In our experiments, we compared Aster against several mainstream real-world graph databases. The results demonstrate that Aster outperforms all baseline graph databases, especially on large-scale graphs. Notably, on the billion-scale Twitter graph dataset, Aster achieves up to 17x throughput improvement compared to the best-performing baseline graph system.