Anatomical Heterogeneity in Transformer Language Models

TL;DR

This paper reveals significant anatomical heterogeneity in transformer language models, showing that different layers have varying importance and training needs, which can be exploited to reduce training costs.

Contribution

It provides empirical evidence of layer heterogeneity in transformer models and introduces a growth training method that allocates resources based on layer importance for efficiency.

Findings

Layer weights follow a regular oscillatory pattern with high predictability.

Layer importance varies over a 10^7 range, affecting model performance.

Growth Transformer Training reduces training costs by ~54% while maintaining quality.

Abstract

Current transformer language models are trained with uniform computational budgets across all layers, implicitly assuming layer homogeneity. We challenge this assumption through empirical analysis of SmolLM2-135M, a 30-layer, 135M-parameter causal language model, using five diagnostic metrics: weight predictability (R2), ablation degradation, recovery speed, weight manipulation robustness, and structural analysis. We find profound anatomical heterogeneity: (1) Layer weights follow strong mathematical regularity (R2 = 0.91) with a universal oscillatory delta pattern (correlation ~= -0.50), yet predicted weights cause catastrophic failure due to nonlinear error accumulation. (2) Layer importance spans a 10^7 range, from a critical core (L8-11, up to +63,419% PPL degradation) to anti-layers (L14, L17) whose removal improves performance. (3) Recovery speed correlates with layer importance, indicating differential training requirements. (4) Only weight scaling (alpha = 0.9) preserves model quality among five tested manipulation strategies. (5) Growth Transformer Training, allocating budget by layer importance, achieves ~54% cost reduction. A proof-of-concept experiment confirms this: 4.7x lower validation loss than uniform training at identical parameter count, while being 13% faster.