ASYMP: Fault-tolerant Mining of Massive Graphs

TL;DR

ASYMP is a scalable, fault-tolerant distributed graph processing system capable of analyzing trillion-edge graphs efficiently, outperforming previous frameworks in speed and scalability at Google.

Contribution

Introduces ASYMP, a novel distributed graph processing system with robust fault tolerance, lockless architecture, and efficient data access, enabling analysis of the largest graphs with improved speed and scalability.

Findings

Runs Connected Component 3-50x faster than MapReduce and Pregel.

Scales linearly with graph size without additional machines.

Fault tolerance: 50% machine failures increase runtime by only 41%.

Abstract

We present ASYMP, a distributed graph processing system developed for the timely analysis of graphs with trillions of edges. ASYMP has several distinguishing features including a robust fault tolerance mechanism, a lockless architecture which scales seamlessly to thousands of machines, and efficient data access patterns to reduce per-machine overhead. ASYMP is used to analyze the largest graphs at Google, and the graphs we consider in our empirical evaluation here are, to the best of our knowledge, the largest considered in the literature. Our experimental results show that compared to previous graph processing frameworks at Google, ASYMP can scale to larger graphs, operate on more crowded clusters, and complete real-world graph mining analytic tasks faster. First, we evaluate the speed of ASYMP, where we show that across a diverse selection of graphs, it runs Connected Component 3-50x faster than state of the art implementations in MapReduce and Pregel. Then we demonstrate the scalability and parallelism of this framework: first by showing that the running time increases linearly by increasing the size of the graphs (without changing the number of machines), and then by showing the gains in running time while increasing the number of machines. Finally, we demonstrate the fault-tolerance properties for the framework, showing that inducing 50% of our machines to fail increases the running time by only 41%.