GX-Plug: a Middleware for Plugging Accelerators to Distributed Graph Processing

TL;DR

GX-Plug is a versatile middleware that effectively integrates accelerators into distributed graph processing systems, significantly enhancing performance and scalability with up to 20x acceleration.

Contribution

It introduces GX-Plug, a middleware supporting various environments and models, with novel optimization techniques for efficient accelerator integration in distributed graph systems.

Findings

Achieves up to 20x acceleration in experiments

Supports multiple runtime and programming models

Enhances scalability and performance of distributed graph systems

Abstract

Recently, research communities highlight the necessity of formulating a scalability continuum for large-scale graph processing, which gains the scale-out benefits from distributed graph systems, and the scale-up benefits from high-performance accelerators. To this end, we propose a middleware, called the GX-plug, for the ease of integrating the merits of both. As a middleware, the GX-plug is versatile in supporting different runtime environments, computation models, and programming models. More, for improving the middleware performance, we study a series of techniques, including pipeline shuffle, synchronization caching and skipping, and workload balancing, for intra-, inter-, and beyond-iteration optimizations, respectively. Experiments show that our middleware efficiently plugs accelerators to representative distributed graph systems, e.g., GraphX and Powergraph, with up-to 20x acceleration ratio.