SiameseNorm: Breaking the Barrier to Reconciling Pre/Post-Norm

TL;DR

SiameseNorm introduces a two-stream Transformer architecture that effectively reconciles the long-standing Pre- and Post-Norm trade-off, enhancing performance and stability across diverse models and modalities.

Contribution

It proposes SiameseNorm, a novel two-stream architecture coupling Pre- and Post-Norm-like pathways, compatible with existing training recipes and applicable across various Transformer-based models.

Findings

Consistently improves performance across language, vision, and diffusion models.

Maintains training stability while enhancing model accuracy.

Effective across models from 400M to 15B parameters.

Abstract

The long-standing tension between Pre- and Post-Norm remains an open problem in Transformer architecture, reflecting a fundamental trade-off between training stability and representational capacity. Prior attempts to combine their strengths have made progress, but often show limited robustness across training settings, restricting their broader applicability. We revisit this dilemma, showing that single-stream architectures struggle to reconcile Pre-Norm's stable identity-gradient propagation with Post-Norm's normalization of the main residual path. To address this structural tension, we propose SiameseNorm, a simple yet effective two-stream architecture that remains compatible with Pre-Norm training recipes. SiameseNorm couples Pre-Norm-like and Post-Norm-like streams through shared residual blocks, allowing each residual block to receive optimization signals from both pathways with negligible overhead. Extensive experiments on 400M and 1.3B dense language models, 15B MoE models, Vision Transformers, and Diffusion Transformers show that SiameseNorm consistently improves performance while maintaining strong training stability across architectures and modalities. Code is available at https://github.com/Qwen-Applications/SiameseNorm.