Test-time Adaptation of Tiny Recursive Models

TL;DR

This paper demonstrates that starting from a pre-trained tiny recursive model and applying full fine-tuning allows efficient adaptation to ARC tasks within compute limits, achieving competitive scores.

Contribution

It introduces a method for effective test-time adaptation of tiny recursive models through full fine-tuning, enabling competitive performance within compute constraints.

Findings

Pre-trained tiny recursive models can be fine-tuned efficiently for ARC tasks.

Full fine-tuning outperforms LoRA or task embedding fine-tuning.

Achieved 6.67% score on semi-private evaluation after limited training steps.

Abstract

Prior to the close of the 2025 ARC Prize competition, the leading open source approach - known as TRM, or Tiny Recursive Models - involved training a 7M parameter recursive neural network on augmented variants of ARC tasks. That approach scored approximately 7.8% on the public ARC AGI II evaluation set, but required a level of compute far in excess of what is allowed during the competition. This paper shows that, by starting from a tiny recursive model that has been pre-trained on public ARC tasks, one can efficiently fine-tune on competition tasks within the allowed compute limits. Specifically, a model was pre-trained on 1,280 public tasks for 700k+ optimizer steps over 48 hours on 4xH100 SXM GPUs to obtain a ~10% score on the public evaluation set. That model was then post-trained in just 12,500 gradient steps during the competition to reach a score of 6.67% on semi-private evaluation tasks. Notably, such post-training performance is achieved by full-fine tuning of the tiny model, not LoRA fine-tuning or fine-tuning of task embeddings alone.