Cost-Effective Proxy Reward Model Construction with On-Policy and Active Learning

TL;DR

This paper proposes a cost-effective method for constructing proxy reward models in reinforcement learning with human feedback, using on-policy queries and active learning to minimize expert labeling costs while improving model performance.

Contribution

It introduces a novel on-policy and active learning strategy for proxy reward model construction that significantly reduces expert query costs in RLHF.

Findings

Achieves over 1% improvement on multiple benchmarks with only 1.7K queries.

Effectively labels nine times more preference pairs with minimal labeled data.

Method is compatible with other expert query strategies for further cost reduction.

Abstract

Reinforcement learning with human feedback (RLHF), as a widely adopted approach in current large language model pipelines, is \textit{bottlenecked by the size of human preference data}. While traditional methods rely on offline preference dataset constructions, recent approaches have shifted towards online settings, where a learner uses a small amount of labeled seed data and a large pool of unlabeled prompts to iteratively construct new preference data through self-generated responses and high-quality reward/preference feedback. However, most current online algorithms still focus on preference labeling during policy model updating with given feedback oracles, which incurs significant expert query costs. \textit{We are the first to explore cost-effective proxy reward oracles construction strategies for further labeling preferences or rewards with extremely limited labeled data and expert query budgets}. Our approach introduces two key innovations: (1) on-policy query to avoid OOD and imbalance issues in seed data, and (2) active learning to select the most informative data for preference queries. Using these methods, we train a evaluation model with minimal expert-labeled data, which then effectively labels nine times more preference pairs for further RLHF training. For instance, our model using Direct Preference Optimization (DPO) gains around over 1% average improvement on AlpacaEval2, MMLU-5shot and MMLU-0shot, with only 1.7K query cost. Our methodology is orthogonal to other direct expert query-based strategies and therefore might be integrated with them to further reduce query costs.