On the emergence of simplex symmetry in the final and penultimate layers of neural network classifiers

TL;DR

This paper investigates the geometric symmetry properties of neural network classifiers, revealing that penultimate layer features form a simplex structure and analyzing conditions under which this symmetry emerges or fails.

Contribution

It provides an analytical explanation for the emergence of simplex symmetry in deep networks and demonstrates when such symmetry appears or breaks down in shallow versus deep models.

Findings

Penultimate layer features form a regular simplex in high dimensions.

Simplex symmetry is analytically explained in toy models of deep networks.

Shallow networks or layers without proper geometric configuration do not exhibit this symmetry.

Abstract

A recent numerical study observed that neural network classifiers enjoy a large degree of symmetry in the penultimate layer. Namely, if $h(x) = Af(x) +b$ where $A$ is a linear map and $f$ is the output of the penultimate layer of the network (after activation), then all data points $x_{i, 1}, \dots, x_{i, N_i}$ in a class $C_i$ are mapped to a single point $y_i$ by $f$ and the points $y_i$ are located at the vertices of a regular $k-1$-dimensional standard simplex in a high-dimensional Euclidean space. We explain this observation analytically in toy models for highly expressive deep neural networks. In complementary examples, we demonstrate rigorously that even the final output of the classifier $h$ is not uniform over data samples from a class $C_i$ if $h$ is a shallow network (or if the deeper layers do not bring the data samples into a convenient geometric configuration).