Efficient Two-Level Scheduling for Concurrent Graph Processing

TL;DR

This paper introduces a two-level scheduling approach for concurrent graph processing that reduces memory redundancy and improves efficiency by leveraging data access correlations and prioritized data management.

Contribution

The paper proposes a novel two-level scheduling strategy that enhances concurrent graph processing by reducing memory redundancy and accelerating convergence.

Findings

Significant improvement over state-of-the-art in processing efficiency.

Reduces memory access redundancy through correlations-aware scheduling.

Accelerates convergence speed of concurrent graph jobs.

Abstract

With the rapidly growing demand of graph processing in the real scene, they have to efficiently handle massive concurrent jobs. Although existing work enable to efficiently handle single graph processing job, there are plenty of memory access redundancy caused by ignoring the characteristic of data access correlations. Motivated such an observation, we proposed two-level scheduling strategy in this paper, which enables to enhance the efficiency of data access and to accelerate the convergence speed of concurrent jobs. Firstly, correlations-aware job scheduling allows concurrent jobs to process the same graph data in Cache, which fundamentally alleviates the challenge of CPU repeatedly accessing the same graph data in memory. Secondly, multiple priority-based data scheduling provides the support of prioritized iteration for concurrent jobs, which is based on the global priority generated by individual priority of each job. Simultaneously, we adopt block priority instead of fine-grained priority to schedule graph data to decrease the computation cost. In particular, two-level scheduling significantly advance over the state-of-the-art because it works in the interlayer between data and systems.