Sparse Growing Transformer: Training-Time Sparse Depth Allocation via Progressive Attention Looping

TL;DR

The paper introduces Sparse Growing Transformer (SGT), a method that dynamically allocates depth during training by selectively looping attention heads, leading to efficiency gains and improved performance over static methods.

Contribution

SGT is a novel training-time sparse depth allocation framework that progressively extends recurrence in Transformers, reducing redundancy and enhancing training efficiency.

Findings

SGT outperforms static block-level looping baselines across multiple scales.

Reduces additional training FLOPs from 16-20% to 1-3%.

Leverages high-entropy attention heads for semantic integration.

Abstract

Existing approaches to increasing the effective depth of Transformers predominantly rely on parameter reuse, extending computation through recursive execution. Under this paradigm, the network structure remains static along the training timeline, and additional computational depth is uniformly assigned to entire blocks at the parameter level. This rigidity across training time and parameter space leads to substantial computational redundancy during training. In contrast, we argue that depth allocation during training should not be a static preset, but rather a progressively growing structural process. Our systematic analysis reveals a deep-to-shallow maturation trajectory across layers, where high-entropy attention heads play a crucial role in semantic integration. Motivated by this observation, we introduce the Sparse Growing Transformer (SGT). SGT is a training-time sparse depth allocation framework that progressively extends recurrence from deeper to shallower layers via targeted attention looping on informative heads. This mechanism induces structural sparsity by selectively increasing depth only for a small subset of parameters as training evolves. Extensive experiments across multiple parameter scales demonstrate that SGT consistently outperforms training-time static block-level looping baselines under comparable settings, while reducing the additional training FLOPs overhead from approximately 16--20% to only 1--3% relative to a standard Transformer backbone.