On the Robustness of Neural Collapse and the Neural Collapse of Robustness

TL;DR

This paper investigates Neural Collapse, revealing its vulnerability to adversarial attacks and its persistence in robust models, and explores how this phenomenon relates to network robustness and generalization.

Contribution

It provides the first analysis of Neural Collapse's stability under adversarial perturbations and its role in robust neural network representations.

Findings

Neural Collapse disappears under small adversarial attacks.

Perturbed examples leap between simplex vertices.

Robust models exhibit aligned simplices in representations.

Abstract

Neural Collapse refers to the curious phenomenon in the end of training of a neural network, where feature vectors and classification weights converge to a very simple geometrical arrangement (a simplex). While it has been observed empirically in various cases and has been theoretically motivated, its connection with crucial properties of neural networks, like their generalization and robustness, remains unclear. In this work, we study the stability properties of these simplices. We find that the simplex structure disappears under small adversarial attacks, and that perturbed examples "leap" between simplex vertices. We further analyze the geometry of networks that are optimized to be robust against adversarial perturbations of the input, and find that Neural Collapse is a pervasive phenomenon in these cases as well, with clean and perturbed representations forming aligned simplices, and giving rise to a robust simple nearest-neighbor classifier. By studying the propagation of the amount of collapse inside the network, we identify novel properties of both robust and non-robust machine learning models, and show that earlier, unlike later layers maintain reliable simplices on perturbed data. Our code is available at https://github.com/JingtongSu/robust_neural_collapse .