Enhancing Reinforcement Learning Fine-Tuning with an Online Refiner

TL;DR

This paper introduces dynamic constraints in reinforcement learning fine-tuning, using an online refiner to adapt constraints based on output quality, leading to improved task rewards and stability.

Contribution

It proposes a novel online refiner mechanism that dynamically adjusts constraints during reinforcement learning fine-tuning, enhancing performance and stability.

Findings

Outperforms KL regularization and unconstrained baselines.

Achieves higher task rewards in dialogue and code generation.

Maintains training stability with dynamic constraints.

Abstract

Constraints are essential for stabilizing reinforcement learning fine-tuning (RFT) and preventing degenerate outputs, yet they inherently conflict with the optimization objective because stronger constraints limit the ability of a fine-tuned model to discover better solutions. We propose \textit{dynamic constraints} that resolve this tension by adapting to the evolving capabilities of the fine-tuned model based on the insight that constraints should only intervene when degenerate outputs occur. We implement this by using a reference model as an \textit{online refiner} that takes the response from the fine-tuned model and generates a minimally corrected version which preserves correct content verbatim while fixing errors. A supervised fine-tuning loss then trains the fine-tuned model to produce the refined output. This mechanism yields a constraint that automatically strengthens or relaxes based on output quality. Experiments on dialogue and code generation show that dynamic constraints outperform both KL regularization and unconstrained baselines, achieving substantially higher task rewards while maintaining training stability.