$\mathbb{X}$-Sample Contrastive Loss: Improving Contrastive Learning with Sample Similarity Graphs

TL;DR

This paper introduces $ ext{X}$-Sample Contrastive Loss, a novel approach that explicitly encodes sample relations beyond binary positives, leading to improved vision representations across various datasets and tasks.

Contribution

The paper proposes $ ext{X}$-Sample Contrastive Loss, which incorporates sample similarities into the contrastive learning framework, enhancing representation quality especially in lower-data regimes.

Findings

Outperforms CLIP on ImageNet and ImageNet Real with 0.6% gain on CC12M.

Achieves significant improvements in low-data regimes, with 16.8% and 18.1% gains on ImageNet and ImageNet Real.

Encourages learning of object, attribute, and background separation in representations.

Abstract

Learning good representations involves capturing the diverse ways in which data samples relate. Contrastive loss - an objective matching related samples - underlies methods from self-supervised to multimodal learning. Contrastive losses, however, can be viewed more broadly as modifying a similarity graph to indicate how samples should relate in the embedding space. This view reveals a shortcoming in contrastive learning: the similarity graph is binary, as only one sample is the related positive sample. Crucially, similarities \textit{across} samples are ignored. Based on this observation, we revise the standard contrastive loss to explicitly encode how a sample relates to others. We experiment with this new objective, called $\mathbb{X}$-Sample Contrastive, to train vision models based on similarities in class or text caption descriptions. Our study spans three scales: ImageNet-1k with 1 million, CC3M with 3 million, and CC12M with 12 million samples. The representations learned via our objective outperform both contrastive self-supervised and vision-language models trained on the same data across a range of tasks. When training on CC12M, we outperform CLIP by $0.6\%$ on both ImageNet and ImageNet Real. Our objective appears to work particularly well in lower-data regimes, with gains over CLIP of $16.8\%$ on ImageNet and $18.1\%$ on ImageNet Real when training with CC3M. Finally, our objective seems to encourage the model to learn representations that separate objects from their attributes and backgrounds, with gains of $3.3$-$5.6$\% over CLIP on ImageNet9. We hope the proposed solution takes a small step towards developing richer learning objectives for understanding sample relations in foundation models.