The Ubiquity of Large Graphs and Surprising Challenges of Graph Processing: Extended Survey

TL;DR

This extensive survey reveals that real-world graph processing involves diverse, large datasets with major challenges in scalability and visualization, and highlights popular applications like data integration and fraud detection.

Contribution

The paper provides the first comprehensive empirical study on practical graph usage, identifying key challenges and application domains in real-world graph processing.

Findings

Graphs are highly diverse and often very large in practice.

Scalability and visualization are the most pressing challenges.

Popular applications include data integration, recommendations, and fraud detection.

Abstract

Graph processing is becoming increasingly prevalent across many application domains. In spite of this prevalence, there is little research about how graphs are actually used in practice. We performed an extensive study that consisted of an online survey of 89 users, a review of the mailing lists, source repositories, and whitepapers of a large suite of graph software products, and in-person interviews with 6 users and 2 developers of these products. Our online survey aimed at understanding: (i) the types of graphs users have; (ii) the graph computations users run; (iii) the types of graph software users use; and (iv) the major challenges users face when processing their graphs. We describe the participants' responses to our questions highlighting common patterns and challenges. Based on our interviews and survey of the rest of our sources, we were able to answer some new questions that were raised by participants' responses to our online survey and understand the specific applications that use graph data and software. Our study revealed surprising facts about graph processing in practice. In particular, real-world graphs represent a very diverse range of entities and are often very large, scalability and visualization are undeniably the most pressing challenges faced by participants, and data integration, recommendations, and fraud detection are very popular applications supported by existing graph software. We hope these findings can guide future research.