Start Late or Finish Early: A Distributed Graph Processing System with Redundancy Reduction

TL;DR

This paper introduces SLFE, a distributed graph processing system that reduces redundant computations by leveraging topological knowledge, significantly improving performance on real-world graphs without sacrificing parallelism.

Contribution

SLFE is a novel distributed graph system with a preprocessing stage and redundancy-aware computation model that effectively minimizes redundant work during graph processing.

Findings

SLFE achieves up to 74.8x speedup over existing systems.

Redundancy reduction schemes are applicable to other vertex-centric systems.

Preprocessing introduces negligible overhead while providing valuable topological insights.

Abstract

Graph processing systems are important in the big data domain. However, processing graphs in parallel often introduces redundant computations in existing algorithms and models. Prior work has proposed techniques to optimize redundancies for the out-of-core graph systems, rather than the distributed graph systems. In this paper, we study various state-of-the-art distributed graph systems and observe root causes for these pervasively existing redundancies. To reduce redundancies without sacrificing parallelism, we further propose SLFE, a distributed graph processing system, designed with the principle of "start late or finish early". SLFE employs a novel preprocessing stage to obtain a graph's topological knowledge with negligible overhead. SLFE's redundancy-aware vertex-centric computation model can then utilize such knowledge to reduce the redundant computations at runtime. SLFE also provides a set of APIs to improve the programmability. Our experiments on an 8-node high-performance cluster show that SLFE outperforms all well-known distributed graph processing systems on real-world graphs (yielding up to 74.8x speedup). SLFE's redundancy-reduction schemes are generally applicable to other vertex-centric graph processing systems.