Learning to Compress: Local Rank and Information Compression in Deep Neural Networks

TL;DR

This paper explores how deep neural networks tend to learn low-dimensional feature representations by reducing local rank during training, linking this behavior to information compression and the Information Bottleneck theory.

Contribution

It introduces the concept of local rank as a measure of feature manifold dimensionality and demonstrates its decrease during training, connecting rank reduction to mutual information compression.

Findings

Local rank decreases during the final training phase.

Rank reduction correlates with mutual information compression.

Theoretical and empirical evidence supports the link between rank and information bottleneck.

Abstract

Deep neural networks tend to exhibit a bias toward low-rank solutions during training, implicitly learning low-dimensional feature representations. This paper investigates how deep multilayer perceptrons (MLPs) encode these feature manifolds and connects this behavior to the Information Bottleneck (IB) theory. We introduce the concept of local rank as a measure of feature manifold dimensionality and demonstrate, both theoretically and empirically, that this rank decreases during the final phase of training. We argue that networks that reduce the rank of their learned representations also compress mutual information between inputs and intermediate layers. This work bridges the gap between feature manifold rank and information compression, offering new insights into the interplay between information bottlenecks and representation learning.