# Uncovering Neural Learning Dynamics Through Latent Mutual Information

## TL;DR

This paper explores how neural networks process information during learning by tracking mutual information, revealing that label-relevant information concentrates in specific channels.

## Contribution

The study introduces a new perspective on representation learning through selective concentration and decorrelation of information.

## Key findings

- Label-relevant mutual information increases with network depth across different architectures.
- High-MI channels are functionally important for accuracy, as confirmed by inference-time experiments.
- A dependence-aware regularizer can improve training speed and accuracy by encouraging desired information patterns.

## Abstract

We study how convolutional neural networks reorganize information during learning in natural image classification tasks by tracking mutual information (MI) between inputs, intermediate representations, and labels. Across VGG-16, ResNet-18, and ResNet-50, we find that label-relevant MI grows reliably with depth while input MI depends strongly on architecture and activation, indicating that “compression’’ is not a universal phenomenon. Within convolutional layers, label information becomes increasingly concentrated in a small subset of channels; inference-time knockouts, shuffles, and perturbations confirm that these high-MI channels are functionally necessary for accuracy. This behavior suggests a view of representation learning driven by selective concentration and decorrelation rather than global information reduction. Finally, we show that a simple dependence-aware regularizer based on the Hilbert–Schmidt Independence Criterion can encourage these same patterns during training, yielding small accuracy gains and consistently faster convergence.