Clustering-based Partitioning for Large Web Graphs

TL;DR

This paper introduces a novel clustering-based streaming algorithm for web graph partitioning that improves quality and scalability, significantly reducing runtime compared to existing methods in large-scale web graph analytics.

Contribution

It proposes a new restreaming vertex-cut partitioning algorithm leveraging web graph clustering, enhancing scalability and efficiency in large-scale graph processing.

Findings

Outperforms state-of-the-art vertex-cut methods on real datasets.

Achieves an order of magnitude lower runtime with many partitions.

Effective in large-scale web graph analytics applications.

Abstract

Graph partitioning plays a vital role in distributedlarge-scale web graph analytics, such as pagerank and labelpropagation. The quality and scalability of partitioning strategyhave a strong impact on such communication- and computation-intensive applications, since it drives the communication costand the workload balance among distributed computing nodes.Recently, the streaming model shows promise in optimizing graphpartitioning. However, existing streaming partitioning strategieseither lack of adequate quality or fall short in scaling with alarge number of partitions.In this work, we explore the property of web graph clusteringand propose a novel restreaming algorithm for vertex-cut parti-tioning. We investigate a series of techniques, which are pipelinedas three steps, streaming clustering, cluster partitioning, andpartition transformation. More, these techniques can be adaptedto a parallel mechanism for further acceleration of partitioning.Experiments on real datasets and real systems show that ouralgorithm outperforms state-of-the-art vertex-cut partitioningmethods in large-scale web graph processing. Surprisingly, theruntime cost of our method can be an order of magnitude lowerthan that of one-pass streaming partitioning algorithms, whenthe number of partitions is large.