Generalization Analysis for Deep Contrastive Representation Learning

TL;DR

This paper derives new generalization bounds for deep contrastive learning that are independent of the number of contrastive tuples and depend on network size and norms, bridging theory gaps.

Contribution

It introduces novel bounds for contrastive learning that avoid exponential depth dependence, using covering numbers and loss augmentation techniques.

Findings

Bounds are independent of tuple size k.

Bounds depend on network size and weight norms.

Techniques bridge contrastive learning and DNN sample complexity theories.

Abstract

In this paper, we present generalization bounds for the unsupervised risk in the Deep Contrastive Representation Learning framework, which employs deep neural networks as representation functions. We approach this problem from two angles. On the one hand, we derive a parameter-counting bound that scales with the overall size of the neural networks. On the other hand, we provide a norm-based bound that scales with the norms of neural networks' weight matrices. Ignoring logarithmic factors, the bounds are independent of $k$, the size of the tuples provided for contrastive learning. To the best of our knowledge, this property is only shared by one other work, which employed a different proof strategy and suffers from very strong exponential dependence on the depth of the network which is due to a use of the peeling technique. Our results circumvent this by leveraging powerful results on covering numbers with respect to uniform norms over samples. In addition, we utilize loss augmentation techniques to further reduce the dependency on matrix norms and the implicit dependence on network depth. In fact, our techniques allow us to produce many bounds for the contrastive learning setting with similar architectural dependencies as in the study of the sample complexity of ordinary loss functions, thereby bridging the gap between the learning theories of contrastive learning and DNNs.