Neural collapse in the orthoplex regime

TL;DR

This paper explores the neural collapse phenomenon in the orthoplex regime, revealing new geometric structures that emerge when the number of classes exceeds the feature space dimension, extending understanding beyond the simplex case.

Contribution

It characterizes the geometric structures of neural collapse in the orthoplex regime, where class count exceeds feature dimension, using Radon's theorem and convexity.

Findings

Neural collapse manifests as orthoplex structures when n > d+1.

The geometric figures differ from the regular simplex observed in the n ≤ d+1 case.

Analysis employs Radon's theorem and convexity to characterize these structures.

Abstract

When training a neural network for classification, the feature vectors of the training set are known to collapse to the vertices of a regular simplex, provided the dimension $d$ of the feature space and the number $n$ of classes satisfies $n\leq d+1$. This phenomenon is known as neural collapse. For other applications like language models, one instead takes $n\gg d$. Here, the neural collapse phenomenon still occurs, but with different emergent geometric figures. We characterize these geometric figures in the orthoplex regime where $d+2\leq n\leq 2d$. The techniques in our analysis primarily involve Radon's theorem and convexity.