Exploring Information-Theoretic Metrics Associated with Neural Collapse in Supervised Training

TL;DR

This paper introduces matrix entropy and new metrics to analyze neural collapse in supervised learning, providing insights into data representation dynamics and enabling improved model fine-tuning.

Contribution

It proposes matrix entropy as an analytical tool, introduces the CMA loss, and develops MIR and HDR metrics to characterize neural network dynamics near Neural Collapse.

Findings

Matrix entropy captures information content variations during Neural Collapse.

MIR and HDR effectively explain training dynamics and phenomena like grokking.

Proposed metrics guide model fine-tuning and understanding neural network behavior.

Abstract

In this paper, we introduce matrix entropy as an analytical tool for studying supervised learning, investigating the information content of data representations and classification head vectors, as well as the dynamic interactions between them during the supervised learning process. Our experimental results reveal that matrix entropy effectively captures the variations in information content of data representations and classification head vectors as neural networks approach Neural Collapse during supervised training, while also serving as a robust metric for measuring similarity among data samples. Leveraging this property, we propose Cross-Model Alignment (CMA) loss to optimize the fine-tuning of pretrained models. To characterize the dynamics of neural networks nearing the Neural Collapse state, we introduce two novel metrics: the Matrix Mutual Information Ratio (MIR) and the Matrix Entropy Difference Ratio (HDR), which quantitatively assess the interactions between data representations and classification heads in supervised learning, with theoretical optimal values derived under the Neural Collapse state. Our experiments demonstrate that MIR and HDR effectively explain various phenomena in neural networks, including the dynamics of standard supervised training, linear mode connectivity. Moreover, we use MIR and HDR to analyze the dynamics of grokking, which is a fascinating phenomenon in supervised learning where a model unexpectedly exhibits generalization long after achieving training data fit.