Towards the Generalization of Multi-view Learning: An Information-theoretical Analysis

TL;DR

This paper develops information-theoretic bounds for multi-view learning, highlighting the importance of capturing both consensus and complementary information to improve generalization, and introduces tighter, data-dependent bounds with faster convergence rates.

Contribution

It provides the first information-theoretic generalization bounds for multi-view learning, emphasizing the role of consensus and complementarity, and introduces novel bounds under various settings with improved rates.

Findings

Bounds show the importance of capturing both consensus and complementary information.

Applying the multi-view information bottleneck regularizer improves generalization.

Numerical results confirm a strong correlation between bounds and true generalization gaps.

Abstract

Multiview learning has drawn widespread attention for its efficacy in leveraging cross-view consensus and complementarity information to achieve a comprehensive representation of data. While multi-view learning has undergone vigorous development and achieved remarkable success, the theoretical understanding of its generalization behavior remains elusive. This paper aims to bridge this gap by developing information-theoretic generalization bounds for multi-view learning, with a particular focus on multi-view reconstruction and classification tasks. Our bounds underscore the importance of capturing both consensus and complementary information from multiple different views to achieve maximally disentangled representations. These results also indicate that applying the multi-view information bottleneck regularizer is beneficial for satisfactory generalization performance. Additionally, we derive novel data-dependent bounds under both leave-one-out and supersample settings, yielding computational tractable and tighter bounds. In the interpolating regime, we further establish the fast-rate bound for multi-view learning, exhibiting a faster convergence rate compared to conventional square-root bounds. Numerical results indicate a strong correlation between the true generalization gap and the derived bounds across various learning scenarios.