Provable Training for Graph Contrastive Learning

TL;DR

This paper introduces a theoretically grounded regularization method called Provable Training (POT) for Graph Contrastive Learning, which improves training balance and embedding quality by identifying and guiding poorly trained nodes.

Contribution

It proposes the node compactness metric, derives its theoretical form, and develops POT to enhance GCL training with provable guarantees and practical improvements.

Findings

POT consistently improves GCL performance across benchmarks.

Node compactness effectively identifies poorly trained nodes.

The method serves as a plug-in to existing GCL approaches.

Abstract

Graph Contrastive Learning (GCL) has emerged as a popular training approach for learning node embeddings from augmented graphs without labels. Despite the key principle that maximizing the similarity between positive node pairs while minimizing it between negative node pairs is well established, some fundamental problems are still unclear. Considering the complex graph structure, are some nodes consistently well-trained and following this principle even with different graph augmentations? Or are there some nodes more likely to be untrained across graph augmentations and violate the principle? How to distinguish these nodes and further guide the training of GCL? To answer these questions, we first present experimental evidence showing that the training of GCL is indeed imbalanced across all nodes. To address this problem, we propose the metric "node compactness", which is the lower bound of how a node follows the GCL principle related to the range of augmentations. We further derive the form of node compactness theoretically through bound propagation, which can be integrated into binary cross-entropy as a regularization. To this end, we propose the PrOvable Training (POT) for GCL, which regularizes the training of GCL to encode node embeddings that follows the GCL principle better. Through extensive experiments on various benchmarks, POT consistently improves the existing GCL approaches, serving as a friendly plugin.