COPR: Continual Human Preference Learning via Optimal Policy Regularization

TL;DR

This paper introduces COPR, a novel continual learning method for reinforcement learning from human feedback that prevents catastrophic forgetting by regularizing policies based on optimal policy theory, improving alignment with human preferences.

Contribution

COPR is the first method to integrate optimal policy regularization with continual learning for RLHF, effectively maintaining historical preferences and enhancing model alignment.

Findings

COPR outperforms strong continual learning baselines in reward and human evaluations.

COPR demonstrates robustness across various settings and model architectures.

Formal proof of learnability supports the theoretical foundation of COPR.

Abstract

Reinforcement Learning from Human Feedback (RLHF) is commonly utilized to improve the alignment of Large Language Models (LLMs) with human preferences. Given the evolving nature of human preferences, continual alignment becomes more crucial and practical in comparison to traditional static alignment. Nevertheless, making RLHF compatible with Continual Learning (CL) is challenging due to its complex process. Meanwhile, directly learning new human preferences may lead to Catastrophic Forgetting (CF) of historical preferences, resulting in helpless or harmful outputs. To overcome these challenges, we propose the Continual Optimal Policy Regularization (COPR) method, which draws inspiration from the optimal policy theory. COPR utilizes a sampling distribution as a demonstration and regularization constraints for CL. It adopts the Lagrangian Duality (LD) method to dynamically regularize the current policy based on the historically optimal policy, which prevents CF and avoids over-emphasizing unbalanced objectives. We also provide formal proof for the learnability of COPR. The experimental results show that COPR outperforms strong CL baselines on our proposed benchmark, in terms of reward-based, GPT-4 evaluations and human assessment. Furthermore, we validate the robustness of COPR under various CL settings, including different backbones, replay memory sizes, and learning orders.