Deep Deterministic Information Bottleneck with Matrix-based Entropy Functional

TL;DR

This paper introduces a novel deep deterministic information bottleneck method using matrix-based Renyi's entropy, which improves neural network generalization and robustness without relying on variational inference.

Contribution

It presents a new entropy functional for the IB principle that avoids variational inference, enhancing neural network performance and robustness.

Findings

Outperforms variational IB in generalization

Enhances robustness to adversarial attacks

Eliminates need for distribution assumptions

Abstract

We introduce the matrix-based Renyi's $\alpha$-order entropy functional to parameterize Tishby et al. information bottleneck (IB) principle with a neural network. We term our methodology Deep Deterministic Information Bottleneck (DIB), as it avoids variational inference and distribution assumption. We show that deep neural networks trained with DIB outperform the variational objective counterpart and those that are trained with other forms of regularization, in terms of generalization performance and robustness to adversarial attack.Code available at https://github.com/yuxi120407/DIB