Rational Metareasoning for Large Language Models

TL;DR

This paper proposes a metareasoning-based training method for large language models that selectively employs reasoning steps, reducing inference costs by 20-37% while maintaining performance.

Contribution

It introduces a novel reward function and training approach inspired by cognitive science to optimize reasoning efficiency in LLMs.

Findings

Reduces inference costs by 20-37% across models

Maintains task performance comparable to existing methods

Uses a reward function based on the Value of Computation

Abstract

Being prompted to engage in reasoning has emerged as a core technique for using large language models (LLMs), deploying additional inference-time compute to improve task performance. However, as LLMs increase in both size and adoption, inference costs are correspondingly becoming increasingly burdensome. How, then, might we optimize reasoning's cost-performance tradeoff? This work introduces a novel approach based on computational models of metareasoning used in cognitive science, training LLMs to selectively use intermediate reasoning steps only when necessary. We first develop a reward function that incorporates the Value of Computation by penalizing unnecessary reasoning, then use this reward function with Expert Iteration to train the LLM. Compared to few-shot chain-of-thought prompting and STaR, our method significantly reduces inference costs (20-37\% fewer tokens generated across three models) while maintaining task performance across diverse datasets.