Recognizable Information Bottleneck
Yilin Lyu, Xin Liu, Mingyang Song, Xinyue Wang, Yaxin Peng, Tieyong, Zeng, Liping Jing

TL;DR
This paper introduces the Recognizable Information Bottleneck (RIB), a new method that improves representation recognizability and generalization in information bottleneck models by leveraging a practical f-CMI-based regularization technique.
Contribution
It establishes a connection between recognizability of representations and the f-CMI generalization bound, and proposes RIB with a recognizability critic for better regularization.
Findings
RIB effectively regularizes models and improves generalization gap estimation.
Experiments show RIB outperforms existing IB methods on standard datasets.
The recognizability critic enhances the interpretability and robustness of learned representations.
Abstract
Information Bottlenecks (IBs) learn representations that generalize to unseen data by information compression. However, existing IBs are practically unable to guarantee generalization in real-world scenarios due to the vacuous generalization bound. The recent PAC-Bayes IB uses information complexity instead of information compression to establish a connection with the mutual information generalization bound. However, it requires the computation of expensive second-order curvature, which hinders its practical application. In this paper, we establish the connection between the recognizability of representations and the recent functional conditional mutual information (f-CMI) generalization bound, which is significantly easier to estimate. On this basis we propose a Recognizable Information Bottleneck (RIB) which regularizes the recognizability of representations through a recognizability…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAdversarial Robustness in Machine Learning · Domain Adaptation and Few-Shot Learning · Stochastic Gradient Optimization Techniques
