Structures and Techniques for Streaming Dynamic Graph Processing on Decentralized Message-Driven Systems

TL;DR

This paper introduces new structures and techniques for scalable, asynchronous, and decentralized dynamic graph processing using message-driven systems, enabling fine-grain updates and efficient streaming of graph data.

Contribution

It proposes a novel co-designed architecture combining asynchronous active messages, data structures, and programming abstractions for dynamic graph processing on memory-driven systems.

Findings

Demonstrated BFS implementation on the proposed system

Achieved fine-grain updates without recomputing from scratch

Identified challenges and opportunities in streaming dynamic graphs

Abstract

The paper presents structures and techniques aimed towards co-designing scalable asynchronous and decentralized dynamic graph processing for fine-grain memory-driven architectures. It uses asynchronous active messages, in the form of actions that send ``work to data'', with a programming and execution model that allows spawning tasks from within the data-parallelism combined with a data-structure that parallelizes vertex object across many scratchpad memory-coupled cores and yet provides a single programming abstraction to the data object. The graph is constructed by streaming new edges using novel message delivery mechanisms and language constructs that work together to pass data and control using abstraction of actions, continuations and local control objects (LCOs) such as futures. It results in very fine-grain updates to a hierarchical dynamic vertex data structure, which subsequently triggers a user application action to update the results of any previous computation without recomputing from scratch. In our experiments we use BFS to demonstrate our concept design, and document challenges and opportunities.