The Homogeneity Trap: Spectral Collapse in Doubly-Stochastic Deep Networks

TL;DR

This paper uncovers a spectral collapse phenomenon in doubly-stochastic deep networks caused by maximum-entropy bias, which limits feature transformation depth and cannot be fixed by Layer Normalization, revealing a fundamental stability-expressivity trade-off.

Contribution

It identifies the Homogeneity Trap as a spectral degradation in DSM-constrained networks and provides theoretical bounds linking spectral properties to network depth and stability.

Findings

Maximum-entropy bias drives spectral collapse towards uniformity.

Spectral bounds relate  to effective network depth.

Layer Normalization cannot prevent collapse in noise-dominated regimes.

Abstract

Doubly-stochastic matrices (DSM) are increasingly utilized in structure-preserving deep architectures -- such as Optimal Transport layers and Sinkhorn-based attention -- to enforce numerical stability and probabilistic interpretability. In this work, we identify a critical spectral degradation phenomenon inherent to these constraints, termed the Homogeneity Trap. We demonstrate that the maximum-entropy bias, typical of Sinkhorn-based projections, drives the mixing operator towards the uniform barycenter, thereby suppressing the subdominant singular value \sigma_2 and filtering out high-frequency feature components. We derive a spectral bound linking \sigma_2 to the network's effective depth, showing that high-entropy constraints restrict feature transformation to a shallow effective receptive field. Furthermore, we formally demonstrate that Layer Normalization fails to mitigate this collapse in noise-dominated regimes; specifically, when spectral filtering degrades the Signal-to-Noise Ratio (SNR) below a critical threshold, geometric structure is irreversibly lost to noise-induced orthogonal collapse. Our findings highlight a fundamental trade-off between entropic stability and spectral expressivity in DSM-constrained networks.