Deep Variational Information Bottleneck

TL;DR

This paper introduces Deep Variational Information Bottleneck (Deep VIB), a neural network-based method that improves model generalization and robustness by approximating the information bottleneck with a variational approach.

Contribution

It proposes a novel variational approximation to the information bottleneck, enabling neural network training with the reparameterization trick for better regularization.

Findings

Models trained with VIB outperform other regularization methods in generalization.

VIB enhances robustness against adversarial attacks.

Deep VIB effectively leverages neural networks for information bottleneck approximation.

Abstract

We present a variational approximation to the information bottleneck of Tishby et al. (1999). This variational approach allows us to parameterize the information bottleneck model using a neural network and leverage the reparameterization trick for efficient training. We call this method "Deep Variational Information Bottleneck", or Deep VIB. We show that models trained with the VIB objective outperform those that are trained with other forms of regularization, in terms of generalization performance and robustness to adversarial attack.