VISAGNN: Versatile Staleness-Aware Efficient Training on Large-Scale Graphs

TL;DR

VISAGNN introduces a dynamic staleness-aware training method for large-scale GNNs that adaptively mitigates the negative effects of stale historical embeddings, improving accuracy and efficiency.

Contribution

The paper proposes a novel staleness-aware GNN training framework that adaptively incorporates staleness into message passing, loss, and embeddings to enhance large-scale graph learning.

Findings

Outperforms existing methods on large-scale benchmarks.

Achieves faster convergence and higher accuracy.

Effectively reduces bias caused by stale embeddings.

Abstract

Graph Neural Networks (GNNs) have shown exceptional success in graph representation learning and a wide range of real-world applications. However, scaling deeper GNNs poses challenges due to the neighbor explosion problem when training on large-scale graphs. To mitigate this, a promising class of GNN training algorithms utilizes historical embeddings to reduce computation and memory costs while preserving the expressiveness of the model. These methods leverage historical embeddings for out-of-batch nodes, effectively approximating full-batch training without losing any neighbor information-a limitation found in traditional sampling methods. However, the staleness of these historical embeddings often introduces significant bias, acting as a bottleneck that can adversely affect model performance. In this paper, we propose a novel VersatIle Staleness-Aware GNN, named VISAGNN, which dynamically and adaptively incorporates staleness criteria into the large-scale GNN training process. By embedding staleness into the message passing mechanism, loss function, and historical embeddings during training, our approach enables the model to adaptively mitigate the negative effects of stale embeddings, thereby reducing estimation errors and enhancing downstream accuracy. Comprehensive experiments demonstrate the effectiveness of our method in overcoming the staleness issue of existing historical embedding techniques, showcasing its superior performance and efficiency on large-scale benchmarks, along with significantly faster convergence.