Universal Reasoner: A Single, Composable Plug-and-Play Reasoner for Frozen LLMs

TL;DR

Universal Reasoner (UniR) is a modular, plug-and-play reasoning component that enhances frozen large language models' reasoning abilities through composable, task-specific modules trained with verifiable rewards.

Contribution

UniR introduces a novel, additive, modular reasoning framework that can be combined with frozen LLMs, enabling efficient, composable, and cross-domain reasoning capabilities.

Findings

UniR outperforms existing fine-tuning methods on mathematical reasoning tasks.

Modules trained on smaller models effectively guide larger LLMs.

UniR generalizes across domains like vision-language and medical reasoning.

Abstract

Large Language Models (LLMs) have demonstrated remarkable general capabilities, but enhancing skills such as reasoning often demands substantial computational resources and may compromise generalization. While Parameter-Efficient Fine-Tuning (PEFT) methods offer a more resource-conscious alternative, they typically require retraining for each LLM backbone due to architectural dependencies. To address these challenges, we propose Universal Reasoner (UniR)-a modular, composable, and plug-and-play reasoning module that can be used with larger frozen LLMs to provide specialized reasoning capabilities with a shared or aligned token space. Specifically, UniR decomposes the reward into a standalone reasoning module trained in a decoupled manner using verifiable rewards, effectively translating trajectory-level signals into token-level guidance. Once trained, UniR is combined with frozen LLMs at inference time by simply adding its output logits to those of the backbone. This additive structure enables modular composition: multiple UniR modules trained for different tasks can be jointly applied by summing their logits, enabling complex reasoning via composition. Furthermore, UniR demonstrates weak-to-strong generalization, where reasoning modules trained on smaller models effectively guide much larger LLMs in the same model family, and generalize across domains such as in vision language models and medical reasoning. Experiments on mathematical reasoning and machine translation show that UniR surpasses existing fine-tuning methods. Code is open-sourced at https://github.com/hangeol/UniR.