Replay Failures as Successes: Sample-Efficient Reinforcement Learning for Instruction Following

TL;DR

This paper introduces HiR, a sample-efficient reinforcement learning framework that reinterprets failed instruction responses as successes to improve instruction-following models with less computational cost.

Contribution

The paper presents HiR, a novel replay strategy that leverages hindsight to turn failures into successes, enhancing RL efficiency for complex instruction following tasks.

Findings

HiR improves instruction-following performance across tasks.

The method reduces computational requirements for RL.

Hindsight replay effectively utilizes binary reward signals.

Abstract

Reinforcement Learning (RL) has shown promise for aligning Large Language Models (LLMs) to follow instructions with various constraints. Despite the encouraging results, RL improvement inevitably relies on sampling successful, high-quality responses; however, the initial model often struggles to generate responses that satisfy all constraints due to its limited capabilities, yielding sparse or indistinguishable rewards that impede learning. In this work, we propose Hindsight instruction Replay (HiR), a novel sample-efficient RL framework for complex instruction following tasks, which employs a select-then-rewrite strategy to replay failed attempts as successes based on the constraints that have been satisfied in hindsight. We perform RL on these replayed samples as well as the original ones, theoretically framing the objective as dual-preference learning at both the instruction- and response-level to enable efficient optimization using only a binary reward signal. Extensive experiments demonstrate that the proposed HiR yields promising results across different instruction following tasks, while requiring less computational budget. Our code and dataset is available at https://github.com/sastpg/HIR.