Universal Reasoning Model

TL;DR

This paper analyzes the sources of performance gains in Universal Transformers for reasoning tasks and introduces the Universal Reasoning Model (URM), which improves reasoning accuracy by adding convolution and truncated backpropagation.

Contribution

The paper identifies the key factors behind UTs' success and proposes URM, a novel model that enhances reasoning performance with simple modifications.

Findings

URM achieves state-of-the-art results on ARC-AGI tasks.

Improvements mainly stem from recurrent bias and nonlinear components, not complex architecture.

URM outperforms previous models with 53.8% pass@1 on ARC-AGI 1.

Abstract

Universal transformers (UTs) have been widely used for complex reasoning tasks such as ARC-AGI and Sudoku, yet the specific sources of their performance gains remain underexplored. In this work, we systematically analyze UTs variants and show that improvements on ARC-AGI primarily arise from the recurrent inductive bias and strong nonlinear components of Transformer, rather than from elaborate architectural designs. Motivated by this finding, we propose the Universal Reasoning Model (URM), which enhances the UT with short convolution and truncated backpropagation. Our approach substantially improves reasoning performance, achieving state-of-the-art 53.8% pass@1 on ARC-AGI 1 and 16.0% pass@1 on ARC-AGI 2. Our code is avaliable at https://github.com/UbiquantAI/URM.