# Nonlinear Information Bottleneck

## TL;DR

This paper introduces a nonlinear information bottleneck method that can handle arbitrary data distributions and nonlinear encoding/decoding, improving over existing variational approaches.

## Contribution

It proposes a novel non-parametric mutual information bound and neural network implementation for nonlinear information bottleneck applicable to diverse data types.

## Key findings

- Outperforms variational IB on real-world datasets
- Handles arbitrary discrete and continuous data distributions
- Supports nonlinear encoding and decoding maps

## Abstract

Information bottleneck (IB) is a technique for extracting information in one random variable $X$ that is relevant for predicting another random variable $Y$. IB works by encoding $X$ in a compressed "bottleneck" random variable $M$ from which $Y$ can be accurately decoded. However, finding the optimal bottleneck variable involves a difficult optimization problem, which until recently has been considered for only two limited cases: discrete $X$ and $Y$ with small state spaces, and continuous $X$ and $Y$ with a Gaussian joint distribution (in which case optimal encoding and decoding maps are linear). We propose a method for performing IB on arbitrarily-distributed discrete and/or continuous $X$ and $Y$, while allowing for nonlinear encoding and decoding maps. Our approach relies on a novel non-parametric upper bound for mutual information. We describe how to implement our method using neural networks. We then show that it achieves better performance than the recently-proposed "variational IB" method on several real-world datasets.