Nexusformer: Nonlinear Attention Expansion for Stable and Inheritable Transformer Scaling

TL;DR

Nexusformer introduces a nonlinear attention expansion method that enables stable, inheritable scaling of Transformer models with less training compute, by overcoming linear projection limitations.

Contribution

It replaces linear Q/K/V projections with a Nexus-Rank layer, allowing lossless structured growth and stable incremental capacity addition.

Findings

Matches Tokenformer's perplexity with 41.5% less compute

Enables lossless growth via zero-initialized blocks

Derives a geometric scaling law for performance prediction

Abstract

Scaling Transformers typically necessitates training larger models from scratch, as standard architectures struggle to expand without discarding learned representations. We identify the primary bottleneck in the attention mechanism's linear projections, which strictly confine feature extraction to fixed-dimensional subspaces, limiting both expressivity and incremental capacity. To address this, we introduce Nexusformer, which replaces linear $Q/K/V$ projections with a Nexus-Rank layer, a three-stage nonlinear mapping driven by dual activations in progressively higher dimensional spaces. This design overcomes the linearity constraint and enables lossless structured growth: new capacity can be injected along two axes via zero-initialized blocks that preserve pretrained knowledge. Experiments on language modeling and reasoning benchmarks demonstrate that Nexusformer matches Tokenformer's perplexity using up to 41.5\% less training compute during progressive scaling (240M to 440M). Furthermore, our analysis of growth dynamics reveals that zero initialization induces a stable convergence trajectory, allowing us to derive a geometric scaling law that accurately predicts performance across expansion scales.