Efficient Online RFT with Plug-and-Play LLM Judges: Unlocking State-of-the-Art Performance

TL;DR

This paper introduces a cost-effective, plug-and-play LLM-based judge that replaces heavyweight models in RLHF, achieving state-of-the-art performance and high interpretability with minimal additional parameters.

Contribution

It presents a novel method using a frozen instruction-tuned 7B LLM with a tiny LoRA adapter as an effective, transparent reward model for RLHF, eliminating the offline tuning phase.

Findings

Achieves 96.2% accuracy on RewardBench, outperforming larger reward networks.

Enables a 7B actor to surpass a 70B baseline in GSM-8K accuracy.

LoRA judge attains 9/10 similarity to human explanations in GPT-4 scoring.

Abstract

Reward-model training is the cost bottleneck in modern Reinforcement Learning Human Feedback (RLHF) pipelines, often requiring tens of billions of parameters and an offline preference-tuning phase. In the proposed method, a frozen, instruction-tuned 7B LLM is augmented with only a one line JSON rubric and a rank-16 LoRA adapter (affecting just 0.8% of the model's parameters), enabling it to serve as a complete substitute for the previously used heavyweight evaluation models. The plug-and-play judge achieves 96.2% accuracy on RewardBench, outperforming specialized reward networks ranging from 27B to 70B parameters. Additionally, it allows a 7B actor to outperform the top 70B DPO baseline, which scores 61.8%, by achieving 92% exact match accuracy on GSM-8K utilizing online PPO. Thorough ablations indicate that (i) six in context demonstrations deliver the majority of the zero-to-few-shot improvements (+2pp), and (ii) the LoRA effectively addresses the remaining disparity, particularly in the safety and adversarial Chat-Hard segments. The proposed model introduces HH-Rationales, a subset of 10,000 pairs from Anthropic HH-RLHF, to examine interpretability, accompanied by human generated justifications. GPT-4 scoring indicates that our LoRA judge attains approximately = 9/10 in similarity to human explanations, while zero-shot judges score around =5/10. These results indicate that the combination of prompt engineering and tiny LoRA produces a cost effective, transparent, and easily adjustable reward function, removing the offline phase while achieving new state-of-the-art outcomes for both static evaluation and online RLHF.