Policy Filtration for RLHF to Mitigate Noise in Reward Models

TL;DR

This paper introduces Policy Filtration for Proximal Policy Optimization (PF-PPO), a method that filters unreliable reward samples in RLHF to improve policy learning, achieving state-of-the-art results in code generation and math reasoning tasks.

Contribution

The paper proposes a novel reward filtering strategy using R2 to enhance RLHF training, significantly improving performance on complex reasoning benchmarks.

Findings

PF-PPO achieves state-of-the-art results on HumanEval, MBPP, and LeetCode.

Filtering unreliable rewards improves policy learning in code and math tasks.

Extensive experiments validate the effectiveness of PF-PPO across multiple benchmarks.

Abstract

While direct policy optimization methods exist, pioneering LLMs are fine-tuned with reinforcement learning from human feedback (RLHF) to generate better responses under the supervision of a reward model learned from preference data. One major challenge of RLHF is the inaccuracy of the intermediate reward model, especially in the tasks that requires complex reasoning for the reward model to score a response. We find that the reliability of the reward model varies across responses assigned with different rewards. This motivates us to filter the samples whose rewards may be unreliable to improve the signal-to-noise ratio during policy learning, resulting in Policy Filtration for Proximal Policy Optimization (PF-PPO). To choose a proper policy filtering strategy, we use the coefficient of determination (R2) between the rewards and actual scores on filtered samples as the metrics to help us find promising strategies since it measures how well the rewards filtered by PF-PPO indicate real performance. We provide extensive experiments to validate the effectiveness of PF-PPO in code generation and math reasoning tasks. In code generation, PF-PPO achieves the state-of-the-art performance of 7-billion-parameter models on HumanEval (+7.9%), MBPP (+0.7%), and LeetCode Contest (+10.0%) which is a more challenging benchmark created by us. In math reasoning, PF-PPO yields performance increase using different reward models and benchmarks (Ape210K and CMATH). Code is available on https://github.com/DtYXs/verl/tree/pf-ppo.