Flag Varieties: A Geometric Framework for Deep Network Alignment

TL;DR

This paper develops a geometric framework using flag varieties to explain layerwise alignment phenomena in deep networks, linking alignment to invariant geometric structures and dynamics.

Contribution

It introduces a unified geometric theory of alignment in deep networks, connecting subspace intersection metrics to flag varieties and explaining neural collapse hierarchies from first principles.

Findings

Alignment geometry is characterized by a flag variety structure.

Weight decay drives exponential subspace alignment.

Nonlinear activations create a commutator obstruction to basis alignment.

Abstract

Alignment, the tendency of adjacent weight matrices in deep networks to develop compatible subspace orientations, underlies gradient flow, Neural Collapse, and representation similarity across architectures. Despite extensive empirical documentation, these phenomena have resisted unified theoretical treatment: existing explanations are post-hoc, each fitted to a specific observation with whatever mathematics is at hand. We reverse this direction by deriving the mathematical structure that layerwise alignment inherently demands. Using geometric invariant theory, we prove that alignment geometry has a canonical closed, polystable stratum given by a flag variety, and that subspace intersection dimension is its unique reparameterization-invariant observable, establishing that subspace metrics are not empirical conventions but mathematical necessities. This unified framework yields two dynamical consequences: ridge regularization drives subspace alignment at an exponential rate set by weight decay, whereas nonlinear activations induce a commutator obstruction to exact basis alignment, generically present in nonlinear networks and absent in linear ones. Together these give a geometric explanation of the Level-2/3 hierarchy in Neural Collapse from first principles rather than post-hoc analysis. The commutator magnitude and head subspace overlap further serve as weight-space windows into internal alignment structure, requiring no forward passes. Experiments on multilayer perceptrons, residual networks, and pretrained language models support the proposed diagnostics and delineate their scope.