Augmentations in Graph Contrastive Learning: Current Methodological Flaws & Towards Better Practices

TL;DR

This paper critically examines current graph contrastive learning methods, revealing that domain-agnostic augmentations can harm representation quality, and proposes task-aware augmentation strategies that significantly improve performance.

Contribution

It identifies flaws in existing augmentation practices for graph CL and introduces effective task-aware augmentation strategies with demonstrated improvements.

Findings

Domain-agnostic augmentations can destroy task-relevant information.

Inductive bias of GNNs can compensate for weak discriminability on small datasets.

Task-aware augmentations improve accuracy by up to 20% in graph classification.

Abstract

Unsupervised graph representation learning is critical to a wide range of applications where labels may be scarce or expensive to procure. Contrastive learning (CL) is an increasingly popular paradigm for such settings and the state-of-the-art in unsupervised visual representation learning. Recent work attributes the success of visual CL to use of task-relevant augmentations and large, diverse datasets. Interestingly, graph CL frameworks report strong performance despite using orders of magnitude smaller datasets and employing domain-agnostic graph augmentations (DAGAs). Motivated by this discrepancy, we probe the quality of representations learnt by popular graph CL frameworks using DAGAs. We find that DAGAs can destroy task-relevant information and harm the model's ability to learn discriminative representations. On small benchmark datasets, we show the inductive bias of graph neural networks can significantly compensate for this weak discriminability. Based on our findings, we propose several sanity checks that enable practitioners to quickly assess the quality of their model's learned representations. We further propose a broad strategy for designing task-aware augmentations that are amenable to graph CL and demonstrate its efficacy on two large-scale, complex graph applications. For example, in graph-based document classification, we show task-aware augmentations improve accuracy up to 20%.