K-Core Decomposition on Super Large Graphs with Limited Resources

TL;DR

This paper introduces a divide-and-conquer approach to improve the efficiency and stability of distributed K-core decomposition on super-large graphs, enabling analysis of graphs with billions of edges under limited resources.

Contribution

It proposes a novel divide-and-conquer strategy that enhances resource efficiency and stability of distributed K-core decomposition on extremely large graphs.

Findings

Significantly reduces resource consumption.

Increases stability of the decomposition process.

Scales to graphs with over 136 billion edges.

Abstract

K-core decomposition is a commonly used metric to analyze graph structure or study the relative importance of nodes in complex graphs. Recent years have seen rapid growth in the scale of the graph, especially in industrial settings. For example, our industrial partner runs popular social applications with billions of users and is able to gather a rich set of user data. As a result, applying K-core decomposition on large graphs has attracted more and more attention from academics and the industry. A simple but effective method to deal with large graphs is to train them in the distributed settings, and some distributed K-core decomposition algorithms are also proposed. Despite their effectiveness, we experimentally and theoretically observe that these algorithms consume too many resources and become unstable on super-large-scale graphs, especially when the given resources are limited. In this paper, we deal with those super-large-scale graphs and propose a divide-and-conquer strategy on top of the distributed K-core decomposition algorithm. We evaluate our approach on three large graphs. The experimental results show that the consumption of resources can be significantly reduced, and the calculation on large-scale graphs becomes more stable than the existing methods. For example, the distributed K-core decomposition algorithm can scale to a large graph with 136 billion edges without losing correctness with our divide-and-conquer technique.