CLUB: A Contrastive Log-ratio Upper Bound of Mutual Information

TL;DR

This paper introduces CLUB, a novel contrastive upper bound for mutual information, enabling effective MI minimization in high-dimensional spaces, with theoretical analysis and practical validation in domain adaptation and information bottleneck tasks.

Contribution

The paper proposes a new contrastive upper bound for mutual information, addressing limitations of existing lower bounds and enabling efficient MI minimization in complex scenarios.

Findings

CLUB reliably estimates mutual information in Gaussian distributions.

The method effectively minimizes MI in domain adaptation tasks.

It accelerates MI minimization with a negative sampling strategy.

Abstract

Mutual information (MI) minimization has gained considerable interests in various machine learning tasks. However, estimating and minimizing MI in high-dimensional spaces remains a challenging problem, especially when only samples, rather than distribution forms, are accessible. Previous works mainly focus on MI lower bound approximation, which is not applicable to MI minimization problems. In this paper, we propose a novel Contrastive Log-ratio Upper Bound (CLUB) of mutual information. We provide a theoretical analysis of the properties of CLUB and its variational approximation. Based on this upper bound, we introduce a MI minimization training scheme and further accelerate it with a negative sampling strategy. Simulation studies on Gaussian distributions show the reliable estimation ability of CLUB. Real-world MI minimization experiments, including domain adaptation and information bottleneck, demonstrate the effectiveness of the proposed method. The code is at https://github.com/Linear95/CLUB.