Mean Mode Screaming: Mean--Variance Split Residuals for 1000-Layer Diffusion Transformers

TL;DR

This paper identifies a collapse mechanism in very deep diffusion transformers caused by mean-dominated homogenization, and proposes a residual modification to enable stable training at 1000 layers.

Contribution

It introduces Mean Mode Screaming (MMS) as a collapse trigger and proposes Mean-Variance Split (MV-Split) Residuals to prevent it, enabling stable training of 1000-layer DiTs.

Findings

MV-Split prevents divergence in 400-layer DiT.

The 1000-layer DiT remains trainable without collapse.

MV-Split outperforms token-isotropic gating methods.

Abstract

Scaling Diffusion Transformers (DiTs) to hundreds of layers introduces a structural vulnerability: networks can enter a silent, mean-dominated collapse state that homogenizes token representations and suppresses centered variation. Through mechanistic auditing, we isolate the trigger event of this collapse as Mean Mode Screaming (MMS). MMS can occur even when training appears stable, with a mean-coherent backward shock on residual writers that opens deep residual branches and drives the network into a mean-dominated state. We show this behavior is driven by an exact decomposition of these gradients into mean-coherent and centered components, compounded by the structural suppression of attention-logit gradients through the null space of the Softmax Jacobian once values homogenize. To address this, we propose Mean-Variance Split (MV-Split) Residuals, which combine a separately gained centered residual update with a leaky trunk-mean replacement. On a 400-layer single-stream DiT, MV-Split prevents the divergent collapse that crashes the un-stabilized baseline; it tracks close to the baseline's pre-crash trajectory while remaining substantially better than token-isotropic gating methods such as LayerScale across the full schedule. Finally, we present a 1000-layer DiT as a scale-validation run at boundary scales, establishing that the architecture remains stably trainable at extreme depth.