Solving the Inverse Alignment Problem for Efficient RLHF

TL;DR

This paper introduces the inverse alignment problem in RLHF, proposing a method to improve reward models by fine-tuning on policy-aligned preference data, leading to better alignment and faster convergence.

Contribution

It formulates the inverse alignment problem for reward model training and demonstrates that fine-tuning on aligned preference data enhances RLHF performance.

Findings

Fine-tuning reward models on aligned preference data improves alignment.

The proposed method achieves faster convergence in RLHF.

Aligned reward models outperform out-of-distribution models.

Abstract

Collecting high-quality preference datasets for reinforcement learning from human feedback (RLHF) is resource-intensive and challenging. As a result, researchers often train reward models on extensive offline datasets which aggregate diverse generation sources and scoring/alignment policies. We hypothesize that this aggregation has an averaging effect on reward model scores, which limits signal and impairs the alignment process. Inspired by the field of inverse RL, we define the 'inverse alignment problem' in language model training, where our objective is to optimize the critic's reward for a fixed actor and a fixed offline preference dataset. We hypothesize that solving the inverse alignment problem will improve reward model quality by providing clearer feedback on the policy's current behavior. To that end, we investigate whether repeatedly fine-tuning a reward model on subsets of the offline preference dataset aligned with a periodically frozen policy during RLHF improves upon vanilla RLHF. Our empirical results demonstrate that this approach facilitates superior alignment and faster convergence compared to using an unaligned or out-of-distribution reward model relative to the LLM policy.