Why Deep Jacobian Spectra Separate: Depth-Induced Scaling and Singular-Vector Alignment

TL;DR

This paper investigates the spectral properties of deep Jacobians, revealing how depth induces exponential scaling and alignment of singular vectors, which explains the implicit bias in deep network training.

Contribution

It introduces a theoretical framework linking depth-induced spectral phenomena to implicit bias, supported by empirical validation in fixed-gates models.

Findings

Depth causes exponential scaling of singular values.

Strong spectral separation leads to singular-vector alignment.

Results support a mechanistic understanding of low-rank Jacobian structures.

Abstract

Understanding why gradient-based training in deep networks exhibits strong implicit bias remains challenging, in part because tractable singular-value dynamics are typically available only for balanced deep linear models. We propose an alternative route based on two theoretically grounded and empirically testable signatures of deep Jacobians: depth-induced exponential scaling of ordered singular values and strong spectral separation. Adopting a fixed-gates view of piecewise-linear networks, where Jacobians reduce to products of masked linear maps within a single activation region, we prove the existence of Lyapunov exponents governing the top singular values at initialization, give closed-form expressions in a tractable masked model, and quantify finite-depth corrections. We further show that sufficiently strong separation forces singular-vector alignment in matrix products, yielding an approximately shared singular basis for intermediate Jacobians. Together, these results motivate an approximation regime in which singular-value dynamics become effectively decoupled, mirroring classical balanced deep-linear analyses without requiring balancing. Experiments in fixed-gates settings validate the predicted scaling, alignment, and resulting dynamics, supporting a mechanistic account of emergent low-rank Jacobian structure as a driver of implicit bias.