Rethinking Minimal Sufficient Representation in Contrastive Learning

TL;DR

This paper reveals that contrastive learning's minimal sufficient representations may omit task-relevant information, leading to performance issues, and proposes a regularization method to incorporate more task-relevant info, improving downstream task performance.

Contribution

It theoretically analyzes the limitations of contrastive learning's minimal representations and introduces a mutual information regularization to enhance task-relevant information inclusion.

Findings

The minimal sufficient representation in contrastive learning is not always sufficient for downstream tasks.

Increasing mutual information between representation and input improves downstream performance.

The proposed method significantly enhances classic contrastive learning models.

Abstract

Contrastive learning between different views of the data achieves outstanding success in the field of self-supervised representation learning and the learned representations are useful in broad downstream tasks. Since all supervision information for one view comes from the other view, contrastive learning approximately obtains the minimal sufficient representation which contains the shared information and eliminates the non-shared information between views. Considering the diversity of the downstream tasks, it cannot be guaranteed that all task-relevant information is shared between views. Therefore, we assume the non-shared task-relevant information cannot be ignored and theoretically prove that the minimal sufficient representation in contrastive learning is not sufficient for the downstream tasks, which causes performance degradation. This reveals a new problem that the contrastive learning models have the risk of over-fitting to the shared information between views. To alleviate this problem, we propose to increase the mutual information between the representation and input as regularization to approximately introduce more task-relevant information, since we cannot utilize any downstream task information during training. Extensive experiments verify the rationality of our analysis and the effectiveness of our method. It significantly improves the performance of several classic contrastive learning models in downstream tasks. Our code is available at https://github.com/Haoqing-Wang/InfoCL.