Multi-View Majority Vote Learning Algorithms: Direct Minimization of PAC-Bayesian Bounds

TL;DR

This paper extends PAC-Bayesian theory to multi-view learning, introducing new bounds based on Rényi divergence and developing algorithms that optimize these bounds for improved ensemble methods.

Contribution

It introduces novel PAC-Bayesian bounds for multi-view learning using Rényi divergence and proposes algorithms to optimize these bounds in practice.

Findings

New Rényi divergence-based generalization bounds for multi-view learning.

Extended C-bound to multi-view settings.

Efficient algorithms aligning with theoretical bounds.

Abstract

The PAC-Bayesian framework has significantly advanced the understanding of statistical learning, particularly for majority voting methods. Despite its successes, its application to multi-view learning -- a setting with multiple complementary data representations -- remains underexplored. In this work, we extend PAC-Bayesian theory to multi-view learning, introducing novel generalization bounds based on R\'enyi divergence. These bounds provide an alternative to traditional Kullback-Leibler divergence-based counterparts, leveraging the flexibility of R\'enyi divergence. Furthermore, we propose first- and second-order oracle PAC-Bayesian bounds and extend the C-bound to multi-view settings. To bridge theory and practice, we design efficient self-bounding optimization algorithms that align with our theoretical results.