A Simple "Try Again" Can Elicit Multi-Turn LLM Reasoning

TL;DR

This paper introduces a simple unary feedback mechanism called UFO for reinforcement learning, which enhances multi-turn reasoning in large language models by enabling better reflection and revision during iterative problem solving.

Contribution

The work presents UFO, a minimal unary feedback approach for multi-turn RL training that improves reasoning accuracy and feedback responsiveness in large language models.

Findings

Multi-turn RL with UFO improves reasoning accuracy by up to 14%.

UFO maintains single-turn performance while enhancing multi-turn reasoning.

Models trained with UFO better reflect and revise answers based on feedback.

Abstract

Multi-turn problem solving is critical yet challenging for Large Reasoning Models (LRMs) to reflect on their reasoning and revise from feedback. Existing Reinforcement Learning (RL) methods train large reasoning models on a single-turn paradigm with verifiable rewards. However, we observe that models trained with existing RL paradigms often lose their ability to solve problems across multiple turns and struggle to revise answers based on contextual feedback, leading to repetitive responses. We ask: can LRMs learn to reflect their answers in a multi-turn context? In this work, we find that training models with multi-turn RL using only unary feedback (e.g., "Let's try again") after wrong answers can improve both single-turn performance and multi-turn reasoning. We introduce Unary Feedback as Observation (UFO) for reinforcement learning, which uses minimal yet common unary user feedback during iterative problem solving. It can be easily applied to existing single-turn RL training setups. Experimental results show that RL training with UFO keeps single-turn performance and improves multi-turn reasoning accuracy by up to 14%, enabling language models to better react to feedback in multi-turn problem solving. To further minimize the number of turns needed for a correct answer while encouraging diverse reasoning when mistakes occur, we design reward structures that guide models to produce careful and deliberate answers in each turn. Code: https://github.com/lichengliu03/unary-feedback