Tiny Autoregressive Recursive Models

TL;DR

This paper investigates the adaptation of Tiny Recursive Models to autoregressive settings, finding that the full autoregressive TRM architecture does not reliably outperform simpler baselines on small tasks, cautioning against its further development.

Contribution

It introduces the Autoregressive TRM and systematically evaluates its performance against simplified models in controlled experiments.

Findings

No consistent performance gains from the full Autoregressive TRM architecture.

Two-level refinement baselines perform strongly on character-level tasks.

Results suggest caution in pursuing autoregressive TRM-specific models.

Abstract

Tiny Recursive Models (TRMs) have recently demonstrated remarkable performance on ARC-AGI, showing that very small models can compete against large foundation models through a two-step refinement mechanism that updates an internal reasoning state $z$ and the predicted output $y$. Naturally, such refinement is of interest for any predictor; it is therefore natural to wonder whether the TRM mechanism could be effectively re-adopted in autoregressive models. However, TRMs cannot be simply compared to standard models because they lack causal predictive structures and contain persistent latent states that make it difficult to isolate specific performance gains. In this paper, we propose the Autoregressive TRM and evaluate it on small autoregressive tasks. To understand its efficacy, we propose a suite of models that gradually transform a standard Transformer to a Tiny Autoregressive Recursive Model in a controlled setting that fixes the block design, token stream, and next-token objective. Across compute-matched experiments on character-level algorithmic tasks, we surprisingly find that there are some two-level refinement baselines that show strong performance. Contrary to expectations, we find no reliable performance gains from the full Autoregressive TRM architecture. These results offer potential promise for two-step refinement mechanisms more broadly but caution against investing in the autoregressive TRM-specific model as a fruitful research direction.