Mutual Information guided Visual Contrastive Learning

TL;DR

This paper introduces a mutual information-based data selection method for contrastive learning, aiming to improve feature generalization by leveraging real-world distribution insights rather than human-designed augmentations.

Contribution

It proposes a novel data augmentation strategy guided by mutual information, enhancing contrastive learning without relying on manual hypotheses or engineering.

Findings

Improved generalization of learned features in open environments.

Effective across multiple state-of-the-art frameworks.

Establishes mutual information as a promising direction for data augmentation.

Abstract

Representation learning methods utilizing the InfoNCE loss have demonstrated considerable capacity in reducing human annotation effort by training invariant neural feature extractors. Although different variants of the training objective adhere to the information maximization principle between the data and learned features, data selection and augmentation still rely on human hypotheses or engineering, which may be suboptimal. For instance, data augmentation in contrastive learning primarily focuses on color jittering, aiming to emulate real-world illumination changes. In this work, we investigate the potential of selecting training data based on their mutual information computed from real-world distributions, which, in principle, should endow the learned features with better generalization when applied in open environments. Specifically, we consider patches attached to scenes that exhibit high mutual information under natural perturbations, such as color changes and motion, as positive samples for learning with contrastive loss. We evaluate the proposed mutual-information-informed data augmentation method on several benchmarks across multiple state-of-the-art representation learning frameworks, demonstrating its effectiveness and establishing it as a promising direction for future research.