Discovering Common Information in Multi-view Data

TL;DR

This paper introduces a mathematically rigorous framework for discovering common and unique information in multi-view data, leveraging information theory and matrix-based entropy estimation, with proven effectiveness on synthetic and real datasets.

Contribution

It proposes a novel supervised multi-view learning framework that explicitly separates common and unique information using total correlation minimization and matrix-based entropy, without variational approximations.

Findings

Outperforms state-of-the-art methods on benchmark datasets

Effectively captures both common and unique information

Provides theoretical guarantees for information discovery

Abstract

We introduce an innovative and mathematically rigorous definition for computing common information from multi-view data, drawing inspiration from G\'acs-K\"orner common information in information theory. Leveraging this definition, we develop a novel supervised multi-view learning framework to capture both common and unique information. By explicitly minimizing a total correlation term, the extracted common information and the unique information from each view are forced to be independent of each other, which, in turn, theoretically guarantees the effectiveness of our framework. To estimate information-theoretic quantities, our framework employs matrix-based R{\'e}nyi's $\alpha$-order entropy functional, which forgoes the need for variational approximation and distributional estimation in high-dimensional space. Theoretical proof is provided that our framework can faithfully discover both common and unique information from multi-view data. Experiments on synthetic and seven benchmark real-world datasets demonstrate the superior performance of our proposed framework over state-of-the-art approaches.