Learning Optimal Multimodal Information Bottleneck Representations

TL;DR

This paper introduces OMIB, a novel multimodal learning framework that guarantees optimal information bottleneck representations by theoretically bounding regularization weights and dynamically balancing modality contributions, leading to improved task performance.

Contribution

The paper proposes OMIB, a new method that ensures optimal multimodal information bottleneck representations with a theoretically grounded regularization scheme and dynamic modality balancing.

Findings

OMIB guarantees the achievability of optimal MIB.

OMIB outperforms state-of-the-art benchmarks in downstream tasks.

Theoretical validation on synthetic data supports OMIB's effectiveness.

Abstract

Leveraging high-quality joint representations from multimodal data can greatly enhance model performance in various machine-learning based applications. Recent multimodal learning methods, based on the multimodal information bottleneck (MIB) principle, aim to generate optimal MIB with maximal task-relevant information and minimal superfluous information via regularization. However, these methods often set ad hoc regularization weights and overlook imbalanced task-relevant information across modalities, limiting their ability to achieve optimal MIB. To address this gap, we propose a novel multimodal learning framework, Optimal Multimodal Information Bottleneck (OMIB), whose optimization objective guarantees the achievability of optimal MIB by setting the regularization weight within a theoretically derived bound. OMIB further addresses imbalanced task-relevant information by dynamically adjusting regularization weights per modality, promoting the inclusion of all task-relevant information. Moreover, we establish a solid information-theoretical foundation for OMIB's optimization and implement it under the variational approximation framework for computational efficiency. Finally, we empirically validate the OMIB's theoretical properties on synthetic data and demonstrate its superiority over the state-of-the-art benchmark methods in various downstream tasks.