Palgol: A High-Level DSL for Vertex-Centric Graph Processing with Remote Data Access

TL;DR

Palgol is a high-level domain-specific language for vertex-centric graph processing that supports remote data access, enabling easier implementation of complex algorithms with dynamic communication in Pregel-like systems.

Contribution

Palgol introduces a declarative syntax with chain access for remote data, and provides a compilation algorithm to generate efficient Pregel code, addressing limitations of existing DSLs.

Findings

Palgol enables implementation of complex Pregel algorithms with dynamic communication.

The efficiency of Palgol-generated code is comparable to hand-written Pregel implementations.

Palgol simplifies programming for large-scale graph algorithms with multiple computation stages.

Abstract

Pregel is a popular distributed computing model for dealing with large-scale graphs. However, it can be tricky to implement graph algorithms correctly and efficiently in Pregel's vertex-centric model, especially when the algorithm has multiple computation stages, complicated data dependencies, or even communication over dynamic internal data structures. Some domain-specific languages (DSLs) have been proposed to provide more intuitive ways to implement graph algorithms, but due to the lack of support for remote access --- reading or writing attributes of other vertices through references --- they cannot handle the above mentioned dynamic communication, causing a class of Pregel algorithms with fast convergence impossible to implement. To address this problem, we design and implement Palgol, a more declarative and powerful DSL which supports remote access. In particular, programmers can use a more declarative syntax called chain access to naturally specify dynamic communication as if directly reading data on arbitrary remote vertices. By analyzing the logic patterns of chain access, we provide a novel algorithm for compiling Palgol programs to efficient Pregel code. We demonstrate the power of Palgol by using it to implement several practical Pregel algorithms, and the evaluation result shows that the efficiency of Palgol is comparable with that of hand-written code.