Invariance-adapted decomposition and Lasso-type contrastive learning

TL;DR

This paper introduces an invariance-adapted latent space decomposition in contrastive learning, generalizing previous work, and demonstrates how Lasso-type metrics can identify these invariant structures, enhancing interpretability and downstream task performance.

Contribution

It proposes a new invariance-adapted latent space decomposition in contrastive learning, extending prior models, and shows how Lasso-type contrastive methods can identify these invariant components.

Findings

Contrastive learning with Lasso-type metrics can find invariance-adapted latent spaces.

The proposed decomposition generalizes previous invariant space models.

Experimental results support the effectiveness of the new decomposition in identifying invariant structures.

Abstract

Recent years have witnessed the effectiveness of contrastive learning in obtaining the representation of dataset that is useful in interpretation and downstream tasks. However, the mechanism that describes this effectiveness have not been thoroughly analyzed, and many studies have been conducted to investigate the data structures captured by contrastive learning. In particular, the recent study of \citet{content_isolate} has shown that contrastive learning is capable of decomposing the data space into the space that is invariant to all augmentations and its complement. In this paper, we introduce the notion of invariance-adapted latent space that decomposes the data space into the intersections of the invariant spaces of each augmentation and their complements. This decomposition generalizes the one introduced in \citet{content_isolate}, and describes a structure that is analogous to the frequencies in the harmonic analysis of a group. We experimentally show that contrastive learning with lasso-type metric can be used to find an invariance-adapted latent space, thereby suggesting a new potential for the contrastive learning. We also investigate when such a latent space can be identified up to mixings within each component.