Generalization Analysis for Supervised Contrastive Representation Learning under Non-IID Settings

TL;DR

This paper analyzes the generalization behavior of Contrastive Representation Learning (CRL) in non-i.i.d. settings, providing theoretical bounds that better reflect practical scenarios where data is recycled across tuples.

Contribution

It introduces the first generalization bounds for CRL under non-i.i.d. conditions, extending theoretical understanding to more realistic data reuse scenarios.

Findings

Generalization bounds scale logarithmically with the class covering number.

Sample complexity depends on the number of classes and feature class complexity.

Bounds are derived for linear and neural network function classes.

Abstract

Contrastive Representation Learning (CRL) has achieved impressive success in various domains in recent years. Nevertheless, the theoretical understanding of the generalization behavior of CRL has remained limited. Moreover, to the best of our knowledge, the current literature only analyzes generalization bounds under the assumption that the data tuples used for contrastive learning are independently and identically distributed. However, in practice, we are often limited to a fixed pool of reusable labeled data points, making it inevitable to recycle data across tuples to create sufficiently large datasets. Therefore, the tuple-wise independence condition imposed by previous works is invalidated. In this paper, we provide a generalization analysis for the CRL framework under non-$i.i.d.$ settings that adheres to practice more realistically. Drawing inspiration from the literature on U-statistics, we derive generalization bounds which indicate that the required number of samples in each class scales as the logarithm of the covering number of the class of learnable feature representations associated to that class. Next, we apply our main results to derive excess risk bounds for common function classes such as linear maps and neural networks.