The Variational Deficiency Bottleneck

TL;DR

This paper introduces the Variational Deficiency Bottleneck, a novel information-theoretic method for learning data representations based on information deficiency, which offers advantages in minimal sufficiency and robustness in classification tasks.

Contribution

It proposes a new bottleneck method grounded in information deficiency, with a variational implementation and operational interpretation, extending the information bottleneck framework.

Findings

Provides a variational upper bound for efficient implementation.

Demonstrates advantages in minimal sufficiency over traditional methods.

Retains robust test performance in classification tasks.

Abstract

We introduce a bottleneck method for learning data representations based on information deficiency, rather than the more traditional information sufficiency. A variational upper bound allows us to implement this method efficiently. The bound itself is bounded above by the variational information bottleneck objective, and the two methods coincide in the regime of single-shot Monte Carlo approximations. The notion of deficiency provides a principled way of approximating complicated channels by relatively simpler ones. We show that the deficiency of one channel with respect to another has an operational interpretation in terms of the optimal risk gap of decision problems, capturing classification as a special case. Experiments demonstrate that the deficiency bottleneck can provide advantages in terms of minimal sufficiency as measured by information bottleneck curves, while retaining robust test performance in classification tasks.