Practice of Streaming Processing of Dynamic Graphs: Concepts, Models, and Systems

TL;DR

This paper provides a comprehensive analysis and taxonomy of dynamic and streaming graph processing systems, clarifying concepts, models, and system architectures to advance understanding and development in this rapidly evolving field.

Contribution

It offers the first detailed taxonomy of streaming graph processing systems, analyzing their architectures, concurrency support, and workloads, and bridges practical systems with theoretical models.

Findings

Identifies key system design patterns for streaming graph processing

Clarifies distinctions among dynamic, temporal, and online graphs

Highlights research challenges and future directions

Abstract

Graph processing has become an important part of various areas of computing, including machine learning, medical applications, social network analysis, computational sciences, and others. A growing amount of the associated graph processing workloads are dynamic, with millions of edges added or removed per second. Graph streaming frameworks are specifically crafted to enable the processing of such highly dynamic workloads. Recent years have seen the development of many such frameworks. However, they differ in their general architectures (with key details such as the support for the concurrent execution of graph updates and queries, or the incorporated graph data organization), the types of updates and workloads allowed, and many others. To facilitate the understanding of this growing field, we provide the first analysis and taxonomy of dynamic and streaming graph processing. We focus on identifying the fundamental system designs and on understanding their support for concurrency, and for different graph updates as well as analytics workloads. We also crystallize the meaning of different concepts associated with streaming graph processing, such as dynamic, temporal, online, and time-evolving graphs, edge-centric processing, models for the maintenance of updates, and graph databases. Moreover, we provide a bridge with the very rich landscape of graph streaming theory by giving a broad overview of recent theoretical related advances, and by discussing which graph streaming models and settings could be helpful in developing more powerful streaming frameworks and designs. We also outline graph streaming workloads and research challenges.