Haste Makes Waste: A Simple Approach for Scaling Graph Neural Networks

TL;DR

This paper introduces REST, a simple and effective training algorithm for scaling graph neural networks that reduces feature staleness, leading to improved performance and faster convergence on large-scale graph benchmarks.

Contribution

The paper proposes REST, a novel training method that mitigates feature staleness in historical embedding-based GNNs, enhancing their scalability and accuracy.

Findings

Achieves 2.7% performance boost on ogbn-papers100M.

Achieves 3.6% performance boost on ogbn-products.

Significantly accelerates convergence of GNN training.

Abstract

Graph neural networks (GNNs) have demonstrated remarkable success in graph representation learning, and various sampling approaches have been proposed to scale GNNs to applications with large-scale graphs. A class of promising GNN training algorithms take advantage of historical embeddings to reduce the computation and memory cost while maintaining the model expressiveness of GNNs. However, they incur significant computation bias due to the stale feature history. In this paper, we provide a comprehensive analysis of their staleness and inferior performance on large-scale problems. Motivated by our discoveries, we propose a simple yet highly effective training algorithm (REST) to effectively reduce feature staleness, which leads to significantly improved performance and convergence across varying batch sizes. The proposed algorithm seamlessly integrates with existing solutions, boasting easy implementation, while comprehensive experiments underscore its superior performance and efficiency on large-scale benchmarks. Specifically, our improvements to state-of-the-art historical embedding methods result in a 2.7% and 3.6% performance enhancement on the ogbn-papers100M and ogbn-products dataset respectively, accompanied by notably accelerated convergence.