Storing and Analyzing Historical Graph Data at Scale

TL;DR

This paper introduces a scalable system for storing and analyzing large volumes of historical graph data, enabling complex temporal queries and analysis of evolving graphs, which was difficult with previous static-focused tools.

Contribution

The paper presents the Historical Graph Store system, including a novel Temporal Graph Index and a Spark-based analysis framework, to efficiently manage and analyze evolving graph data.

Findings

Efficient storage and retrieval of large historical graph data.

Support for complex temporal graph queries.

Scalable analysis of evolving graph properties.

Abstract

The work on large-scale graph analytics to date has largely focused on the study of static properties of graph snapshots. However, a static view of interactions between entities is often an oversimplification of several complex phenomena like the spread of epidemics, information diffusion, formation of online communities}, and so on. Being able to find temporal interaction patterns, visualize the evolution of graph properties, or even simply compare them across time, adds significant value in reasoning over graphs. However, because of lack of underlying data management support, an analyst today has to manually navigate the added temporal complexity of dealing with large evolving graphs. In this paper, we present a system, called Historical Graph Store, that enables users to store large volumes of historical graph data and to express and run complex temporal graph analytical tasks against that data. It consists of two key components: a Temporal Graph Index (TGI), that compactly stores large volumes of historical graph evolution data in a partitioned and distributed fashion; it provides support for retrieving snapshots of the graph as of any timepoint in the past or evolution histories of individual nodes or neighborhoods; and a Spark-based Temporal Graph Analysis Framework (TAF), for expressing complex temporal analytical tasks and for executing them in an efficient and scalable manner. Our experiments demonstrate our system's efficient storage, retrieval and analytics across a wide variety of queries on large volumes of historical graph data.