Augmentation Component Analysis: Modeling Similarity via the Augmentation Overlaps

TL;DR

Augmentation Component Analysis (ACA) introduces a novel way to model sample similarity based on augmentation overlaps, improving embedding quality in self-supervised learning by efficiently capturing semantic relationships.

Contribution

ACA models augmentation distributions as sample descriptions, enabling efficient dimension reduction and similarity preservation without explicit augmentation feature estimation.

Findings

Achieves competitive results on multiple benchmarks.

Provides a theoretical link to PCA for similarity preservation.

Efficiently captures semantic similarity through augmentation overlaps.

Abstract

Self-supervised learning aims to learn a embedding space where semantically similar samples are close. Contrastive learning methods pull views of samples together and push different samples away, which utilizes semantic invariance of augmentation but ignores the relationship between samples. To better exploit the power of augmentation, we observe that semantically similar samples are more likely to have similar augmented views. Therefore, we can take the augmented views as a special description of a sample. In this paper, we model such a description as the augmentation distribution and we call it augmentation feature. The similarity in augmentation feature reflects how much the views of two samples overlap and is related to their semantical similarity. Without computational burdens to explicitly estimate values of the augmentation feature, we propose Augmentation Component Analysis (ACA) with a contrastive-like loss to learn principal components and an on-the-fly projection loss to embed data. ACA equals an efficient dimension reduction by PCA and extracts low-dimensional embeddings, theoretically preserving the similarity of augmentation distribution between samples. Empirical results show our method can achieve competitive results against various traditional contrastive learning methods on different benchmarks.