Efficient block contrastive learning via parameter-free meta-node approximation

TL;DR

This paper introduces a novel, efficient meta-node approximation technique for contrastive learning on graphs that captures all negatives at graph level with linear complexity, improving accuracy and reducing computational costs.

Contribution

The authors propose Proxy approximated meta-node Contrastive (PamC) loss, a scalable, bias-free method that captures all negatives efficiently at graph level, outperforming pair-wise losses.

Findings

Achieves up to 3x faster training time

Reduces GPU memory usage by over 5x

Shows accuracy improvements on 6 benchmark datasets

Abstract

Contrastive learning has recently achieved remarkable success in many domains including graphs. However contrastive loss, especially for graphs, requires a large number of negative samples which is unscalable and computationally prohibitive with a quadratic time complexity. Sub-sampling is not optimal and incorrect negative sampling leads to sampling bias. In this work, we propose a meta-node based approximation technique that can (a) proxy all negative combinations (b) in quadratic cluster size time complexity, (c) at graph level, not node level, and (d) exploit graph sparsity. By replacing node-pairs with additive cluster-pairs, we compute the negatives in cluster-time at graph level. The resulting Proxy approximated meta-node Contrastive (PamC) loss, based on simple optimized GPU operations, captures the full set of negatives, yet is efficient with a linear time complexity. By avoiding sampling, we effectively eliminate sample bias. We meet the criterion for larger number of samples, thus achieving block-contrastiveness, which is proven to outperform pair-wise losses. We use learnt soft cluster assignments for the meta-node constriction, and avoid possible heterophily and noise added during edge creation. Theoretically, we show that real world graphs easily satisfy conditions necessary for our approximation. Empirically, we show promising accuracy gains over state-of-the-art graph clustering on 6 benchmarks. Importantly, we gain substantially in efficiency; up to 3x in training time, 1.8x in inference time and over 5x in GPU memory reduction.