Expanding the Capabilities of Reinforcement Learning via Text Feedback

TL;DR

This paper introduces RL from Text Feedback (RLTF), a framework that leverages human-like textual critiques during training to improve large language models' performance on various tasks, without requiring feedback at inference.

Contribution

It formalizes RLTF, proposes two novel methods (Self Distillation and Feedback Modeling), and demonstrates their effectiveness across multiple benchmarks.

Findings

Both methods outperform strong baselines.

RLTF improves reasoning, math, and creative tasks.

Text feedback is a valuable intermediate supervision source.

Abstract

The success of RL for LLM post-training stems from an unreasonably uninformative source: a single bit of information per rollout as binary reward or preference label. At the other extreme, distillation offers dense supervision but requires demonstrations, which are costly and difficult to scale. We study text feedback as an intermediate signal: richer than scalar rewards, yet cheaper than complete demonstrations. Textual feedback is a natural mode of human interaction and is already abundant in many real-world settings, where users, annotators, and automated judges routinely critique LLM outputs. Towards leveraging text feedback at scale, we formalize a multi-turn RL setup, RL from Text Feedback (RLTF), where text feedback is available during training but not at inference. Therefore, models must learn to internalize the feedback in order to improve their test-time single-turn performance. To do this, we propose two methods: Self Distillation (RLTF-SD), which trains the single-turn policy to match its own feedback-conditioned second-turn generations; and Feedback Modeling (RLTF-FM), which predicts the feedback as an auxiliary objective. We provide theoretical analysis on both methods, and empirically evaluate on reasoning puzzles, competition math, and creative writing tasks. Our results show that both methods consistently outperform strong baselines across benchmarks, highlighting the potential of RL with an additional source of rich supervision at scale.