A Critical Review of Information Bottleneck Theory and its Applications to Deep Learning

TL;DR

This paper critically reviews the information bottleneck theory, an information-theoretic approach that models neural network learning as a trade-off between data compression and information retention, aiming to deepen understanding of deep learning dynamics.

Contribution

It provides a comprehensive overview of IB theory's theoretical foundations and recent applications in analyzing deep neural networks, highlighting current challenges and future directions.

Findings

IB theory offers valuable insights into neural network training dynamics.

Recent applications demonstrate IB's potential in explaining deep learning phenomena.

The review identifies gaps and open problems in applying IB to complex models.

Abstract

In the past decade, deep neural networks have seen unparalleled improvements that continue to impact every aspect of today's society. With the development of high performance GPUs and the availability of vast amounts of data, learning capabilities of ML systems have skyrocketed, going from classifying digits in a picture to beating world-champions in games with super-human performance. However, even as ML models continue to achieve new frontiers, their practical success has been hindered by the lack of a deep theoretical understanding of their inner workings. Fortunately, a known information-theoretic method called the information bottleneck theory has emerged as a promising approach to better understand the learning dynamics of neural networks. In principle, IB theory models learning as a trade-off between the compression of the data and the retainment of information. The goal of this survey is to provide a comprehensive review of IB theory covering it's information theoretic roots and the recently proposed applications to understand deep learning models.