Learning Reward and Policy Jointly from Demonstration and Preference Improves Alignment

TL;DR

This paper introduces a unified approach called AIHF that jointly learns reward models and policies from demonstrations and preferences, improving alignment in large language models and robotic control.

Contribution

The paper presents a single-stage method for joint reward and policy learning that outperforms traditional multi-stage approaches like RLHF and DPO, especially with limited preference data.

Findings

AIHF outperforms RLHF and DPO in experiments.

The approach effectively utilizes limited high-quality preference data.

It simplifies existing alignment pipelines with minor modifications.

Abstract

Aligning human preference and value is an important requirement for building contemporary foundation models and embodied AI. However, popular approaches such as reinforcement learning with human feedback (RLHF) break down the task into successive stages, such as supervised fine-tuning (SFT), reward modeling (RM), and reinforcement learning (RL), each performing one specific learning task. Such a sequential approach results in serious issues such as significant under-utilization of data and distribution mismatch between the learned reward model and generated policy, which eventually lead to poor alignment performance. We develop a single stage approach named Alignment with Integrated Human Feedback (AIHF), capable of integrating both human preference and demonstration to train reward models and the policy. The proposed approach admits a suite of efficient algorithms, which can easily reduce to, and leverage, popular alignment algorithms such as RLHF and Directly Policy Optimization (DPO), and only requires minor changes to the existing alignment pipelines. We demonstrate the efficiency of the proposed solutions with extensive experiments involving alignment problems in LLMs and robotic control problems in MuJoCo. We observe that the proposed solutions outperform the existing alignment algorithms such as RLHF and DPO by large margins, especially when the amount of high-quality preference data is relatively limited.