Towards a Rigorous Analysis of Mutual Information in Contrastive Learning

TL;DR

This paper introduces new methods and theorems to improve the rigor of mutual information analysis in contrastive learning, addressing estimation challenges and clarifying existing misconceptions.

Contribution

It presents three novel methods and related theorems to enhance the accuracy and rigor of mutual information analysis in contrastive learning.

Findings

Reassessed contrastive learning instances with new methods

Clarified misconceptions about mutual information measures

Improved understanding of InfoMin principle

Abstract

Contrastive learning has emerged as a cornerstone in recent achievements of unsupervised representation learning. Its primary paradigm involves an instance discrimination task with a mutual information loss. The loss is known as InfoNCE and it has yielded vital insights into contrastive learning through the lens of mutual information analysis. However, the estimation of mutual information can prove challenging, creating a gap between the elegance of its mathematical foundation and the complexity of its estimation. As a result, drawing rigorous insights or conclusions from mutual information analysis becomes intricate. In this study, we introduce three novel methods and a few related theorems, aimed at enhancing the rigor of mutual information analysis. Despite their simplicity, these methods can carry substantial utility. Leveraging these approaches, we reassess three instances of contrastive learning analysis, illustrating their capacity to facilitate deeper comprehension or to rectify pre-existing misconceptions. Specifically, we investigate small batch size, mutual information as a measure, and the InfoMin principle.