Quantifying the Variability Collapse of Neural Networks

TL;DR

This paper introduces the Variability Collapse Index (VCI), a new metric to quantify neural collapse in last layer features, linking it to transferability and providing theoretical and empirical validation.

Contribution

The paper proposes VCI, a novel, stable metric for quantifying neural collapse, enhancing understanding of transferability in neural networks.

Findings

VCI correlates with neural network transferability.

VCI is invariant under linear transformations.

VCI demonstrates stability and theoretical soundness.

Abstract

Recent studies empirically demonstrate the positive relationship between the transferability of neural networks and the within-class variation of the last layer features. The recently discovered Neural Collapse (NC) phenomenon provides a new perspective of understanding such last layer geometry of neural networks. In this paper, we propose a novel metric, named Variability Collapse Index (VCI), to quantify the variability collapse phenomenon in the NC paradigm. The VCI metric is well-motivated and intrinsically related to the linear probing loss on the last layer features. Moreover, it enjoys desired theoretical and empirical properties, including invariance under invertible linear transformations and numerical stability, that distinguishes it from previous metrics. Our experiments verify that VCI is indicative of the variability collapse and the transferability of pretrained neural networks.