How Much Is One Recurrence Worth? Iso-Depth Scaling Laws for Looped Language Models

TL;DR

This paper investigates the value of recurrence in looped transformers by measuring how shared recurrences compare to unique blocks in terms of validation loss and training compute, introducing a diagnostic exponent.

Contribution

It introduces a scaling law and a recurrence-equivalence exponent to quantify the impact of recurrence sharing versus unique blocks in transformer models.

Findings

The recurrence-equivalence exponent φ is approximately 0.46, indicating partial equivalence between shared recurrences and unique blocks.

Replacing unique blocks with shared recurrences increases validation loss at the same training compute.

Truncated backpropagation lowers φ, showing poorer training of the loop mechanism, while hyperconnections raise φ, indicating capacity gains.

Abstract

We measure how much one recurrence is worth to a looped (depth-recurrent) transformer, in equivalent unique parameters. From an iso-depth pretraining sweep across recurrence counts $r \in \{1, 2, 4, 8\}$ spanning ${\sim}50\times$ in training compute, we fit a joint scaling law $L = E + A\,(N_\text{once} + r^{\varphi} N_\text{rec})^{-\alpha} + B\,D^{-\beta}$ and measure a recurrence-equivalence exponent $\varphi = 0.46$. Intuitively, $\varphi$ tells us whether looping a block $r$ times is equivalent in validation loss to $r$ unique blocks of a non-looped model (full equivalence, $\varphi{=}1$) or to a single block run repeatedly with no capacity gain ($\varphi{=}0$). Our $\varphi = 0.46$ sits in between, so replacing unique blocks with shared recurrences increases validation loss at matched training compute. For example, at $r{=}4$ a 410M looped model performs on par with a 580M non-looped model, but incurs the training cost of a 1B non-looped one. We demonstrate the utility of $\varphi$ as a diagnostic tool on two case studies: commonly used truncated backpropagation lowers $\varphi$ to $0.38$, indicating that the loop mechanism is poorly trained under truncation, even though validation loss decreases. Conversely, hyperconnections raise $\varphi$ to $0.65$, a genuine capacity gain. Our method separates true loop improvements from training-side gains, a distinction raw validation loss cannot make.