Architecture Matters: Uncovering Implicit Mechanisms in Graph Contrastive Learning

TL;DR

This paper systematically studies graph contrastive learning (GCL), revealing that many assumptions from visual contrastive learning do not hold for graphs and emphasizing the importance of graph-specific architecture and implicit biases.

Contribution

It uncovers unique properties of GCL, provides theoretical insights into GNN biases, and advocates for architecture-aware design of GCL methods.

Findings

Positive samples are not essential for GCL.

Negative samples are unnecessary for certain graph tasks with proper normalization.

Simple data augmentations can achieve competitive performance.

Abstract

With the prosperity of contrastive learning for visual representation learning (VCL), it is also adapted to the graph domain and yields promising performance. However, through a systematic study of various graph contrastive learning (GCL) methods, we observe that some common phenomena among existing GCL methods that are quite different from the original VCL methods, including 1) positive samples are not a must for GCL; 2) negative samples are not necessary for graph classification, neither for node classification when adopting specific normalization modules; 3) data augmentations have much less influence on GCL, as simple domain-agnostic augmentations (e.g., Gaussian noise) can also attain fairly good performance. By uncovering how the implicit inductive bias of GNNs works in contrastive learning, we theoretically provide insights into the above intriguing properties of GCL. Rather than directly porting existing VCL methods to GCL, we advocate for more attention toward the unique architecture of graph learning and consider its implicit influence when designing GCL methods. Code is available at https: //github.com/PKU-ML/ArchitectureMattersGCL.