Leveraging Superfluous Information in Contrastive Representation Learning

TL;DR

This paper identifies the presence of superfluous information in contrastive learning and proposes SuperInfo, a new objective that improves downstream task performance by balancing task-relevant and irrelevant information.

Contribution

The paper introduces SuperInfo, a novel contrastive learning objective that explicitly models and leverages superfluous information to enhance representation robustness.

Findings

SuperInfo outperforms traditional contrastive methods on image classification.

The method improves object detection and instance segmentation results.

Tuning loss coefficients effectively discards task-irrelevant information.

Abstract

Contrastive representation learning, which aims to learnthe shared information between different views of unlabeled data by maximizing the mutual information between them, has shown its powerful competence in self-supervised learning for downstream tasks. However, recent works have demonstrated that more estimated mutual information does not guarantee better performance in different downstream tasks. Such works inspire us to conjecture that the learned representations not only maintain task-relevant information from unlabeled data but also carry task-irrelevant information which is superfluous for downstream tasks, thus leading to performance degeneration. In this paper we show that superfluous information does exist during the conventional contrastive learning framework, and further design a new objective, namely SuperInfo, to learn robust representations by a linear combination of both predictive and superfluous information. Besides, we notice that it is feasible to tune the coefficients of introduced losses to discard task-irrelevant information, while keeping partial non-shared task-relevant information according to our SuperInfo loss.We demonstrate that learning with our loss can often outperform the traditional contrastive learning approaches on image classification, object detection and instance segmentation tasks with significant improvements.