Adaptive Label Smoothing To Regularize Large-Scale Graph Training

TL;DR

This paper introduces an adaptive label smoothing method to mitigate label bias in large-scale graph neural network training, improving model calibration and generalization across various frameworks.

Contribution

The proposed ALS method adaptively smooths labels based on neighborhood label distributions, addressing label bias in batch training of large-scale graphs.

Findings

ALS improves generalization performance on real-world datasets.

ALS effectively calibrates biased labels in scalable GNN frameworks.

The method is broadly applicable across different large-scale graph learning models.

Abstract

Graph neural networks (GNNs), which learn the node representations by recursively aggregating information from its neighbors, have become a predominant computational tool in many domains. To handle large-scale graphs, most of the existing methods partition the input graph into multiple sub-graphs (e.g., through node clustering) and apply batch training to save memory cost. However, such batch training will lead to label bias within each batch, and then result in over-confidence in model predictions. Since the connected nodes with positively related labels tend to be assigned together, the traditional cross-entropy minimization process will attend on the predictions of biased classes in the batch, and may intensify the overfitting issue. To overcome the label bias problem, we propose the adaptive label smoothing (ALS) method to replace the one-hot hard labels with smoothed ones, which learns to allocate label confidences from the biased classes to the others. Specifically, ALS propagates node labels to aggregate the neighborhood label distribution in a pre-processing step, and then updates the optimal smoothed labels online to adapt to specific graph structure. Experiments on the real-world datasets demonstrate that ALS can be generally applied to the main scalable learning frameworks to calibrate the biased labels and improve generalization performances.