Learning to Reason from Feedback at Test-Time

TL;DR

This paper introduces FTTT and OpTune, a new framework and optimizer for better feedback utilization in large language models during test-time reasoning, improving scalability and performance.

Contribution

It presents a novel test-time feedback optimization paradigm and a learnable optimizer, addressing limitations of existing methods in feedback utilization for LLMs.

Findings

FTTT and OpTune outperform existing methods in reasoning tasks.

Enhanced scalability and accuracy demonstrated across multiple datasets.

Effective feedback exploitation improves test-time reasoning performance.

Abstract

Solving complex tasks in a single attempt is challenging for large language models (LLMs). Iterative interaction with the environment and feedback is often required to achieve success, making effective feedback utilization a critical topic. Existing approaches either struggle with length generalization or rely on naive retries without leveraging prior information. In this paper, we introduce FTTT, a novel paradigm that formulates feedback utilization as an optimization problem at test time. Additionally, we propose a learnable test-time optimizer, OpTune, to effectively exploit feedback. Experiments on two LLMs across four reasoning datasets demonstrate that FTTT and OpTune achieve superior scalability and performance.