A Feasible Graph Partition Framework for Random Walks Implemented by Parallel Computing in Big Graph

TL;DR

This paper introduces a new graph partition framework optimized for random walks in large graphs, reducing communication costs and balancing load through parallel computing and greedy algorithms.

Contribution

It presents a novel framework with optimization functions and greedy algorithms specifically designed for efficient graph partitioning for random walks in big data environments.

Findings

Communication reduction by over 70 times

Effective load balancing in graph partitions

Framework adaptable to various application needs

Abstract

Graph partition is a fundamental problem of parallel computing for big graph data. Many graph partition algorithms have been proposed to solve the problem in various applications, such as matrix computations and PageRank, etc., but none has pay attention to random walks. Random walks is a widely used method to explore graph structure in lots of fields. The challenges of graph partition for random walks include the large number of times of communication between partitions, lots of replications of the vertices, unbalanced partition, etc. In this paper, we propose a feasible graph partition framework for random walks implemented by parallel computing in big graph. The framework is based on two optimization functions to reduce the bandwidth, memory and storage cost in the condition that the load balance is guaranteed. In this framework, several greedy graph partition algorithms are proposed. We also propose five metrics from different perspectives to evaluate the performance of these algorithms. By running the algorithms on the big graph data set of real world, the experimental results show that these algorithms in the framework are capable of solving the problem of graph partition for random walks for different needs, e.g. the best result is improved more than 70 times in reducing the times of communication.