Logit Arithmetic Elicits Long Reasoning Capabilities Without Training

TL;DR

This paper introduces ThinkLogit, a decoding-time method that elicits long reasoning abilities in large language models without extensive training, using logits arithmetic and a smaller guiding model to significantly improve reasoning performance.

Contribution

Proposes a novel decoding-time approach, ThinkLogit, that elicits long reasoning in large models without training, and enhances it with preference optimization in ThinkLogit-DPO.

Findings

Achieves 26-29% improvement in pass@1 on mathematical datasets.

Demonstrates transfer of reasoning skills via reinforcement learning.

Uses a smaller model as a guide to efficiently elicit reasoning capabilities.

Abstract

Large reasoning models (LRMs) can do complex reasoning via long chain-of-thought (CoT) involving cognitive strategies such as backtracking and self-correction. Recent studies suggest that some models inherently possess these long reasoning abilities, which may be unlocked via extra training. Our work first investigates whether we can elicit such behavior without any training. To this end, we propose a decoding-time approach, ThinkLogit, which utilizes logits arithmetic (Liu et al., 2024) to tune a target large LM for long reasoning using a substantially smaller model as guider. We then show that we can further boost performance by training the guider model with preference optimization over correct/incorrect reasoning pairs sampled from both the target and guider model -- a setup we refer to as ThinkLogit-DPO. Our experiments demonstrate that ThinkLogit and ThinkLogit-DPO achieve a relative improvement in pass@1 by 26% and 29%, respectively, over four mathematical datasets using the Qwen2.5-32B when guided by R1-Distill-Qwen-1.5B -- a model 21x smaller. Lastly, we show that ThinkLogit can transfer long reasoning skills acquired through reinforcement learning, improving pass@1 by 13% relative compared to the Qwen2.5-32B base model. Our work presents a computationally-efficient method to elicit long reasoning in large models with minimal or no additional training.