Revisiting Theory of Contrastive Learning for Domain Generalization

TL;DR

This paper extends the theoretical understanding of contrastive learning by providing new generalization bounds that account for both domain shift and the introduction of new label spaces, addressing real-world domain generalization challenges.

Contribution

It introduces novel theoretical bounds for contrastive learning that explicitly handle distributional shifts and new label spaces in downstream tasks.

Findings

Performance depends on statistical discrepancy between distributions.

Provides guarantees for classification with unseen class distributions.

Addresses both domain shift and new label spaces in theory.

Abstract

Contrastive learning is among the most popular and powerful approaches for self-supervised representation learning, where the goal is to map semantically similar samples close together while separating dissimilar ones in the latent space. Existing theoretical methods assume that downstream task classes are drawn from the same latent class distribution used during the pretraining phase. However, in real-world settings, downstream tasks may not only exhibit distributional shifts within the same label space but also introduce new or broader label spaces, leading to domain generalization challenges. In this work, we introduce novel generalization bounds that explicitly account for both types of mismatch: domain shift and domain generalization. Specifically, we analyze scenarios where downstream tasks either (i) draw classes from the same latent class space but with shifted distributions, or (ii) involve new label spaces beyond those seen during pretraining. Our analysis reveals how the performance of contrastively learned representations depends on the statistical discrepancy between pretraining and downstream distributions. This extended perspective allows us to derive provable guarantees on the performance of learned representations on average classification tasks involving class distributions outside the pretraining latent class set.