Robust Reinforcement Learning from Corrupted Human Feedback

TL;DR

This paper introduces R^3M, a robust reinforcement learning from human feedback method that effectively handles corrupted preference data by modeling outliers, improving reward learning robustness in AI systems.

Contribution

The paper proposes R^3M, a novel robust RLHF approach that models corrupted labels as outliers and provides theoretical guarantees for reward recovery and outlier detection.

Findings

R^3M improves robustness in robotic control tasks.

R^3M enhances natural language generation with large language models.

The method effectively identifies outliers in preference data.

Abstract

Reinforcement learning from human feedback (RLHF) provides a principled framework for aligning AI systems with human preference data. For various reasons, e.g., personal bias, context ambiguity, lack of training, etc, human annotators may give incorrect or inconsistent preference labels. To tackle this challenge, we propose a robust RLHF approach -- $R^3M$, which models the potentially corrupted preference label as sparse outliers. Accordingly, we formulate the robust reward learning as an $\ell_1$-regularized maximum likelihood estimation problem. Computationally, we develop an efficient alternating optimization algorithm, which only incurs negligible computational overhead compared with the standard RLHF approach. Theoretically, we prove that under proper regularity conditions, $R^3M$ can consistently learn the underlying reward and identify outliers, provided that the number of outlier labels scales sublinearly with the preference sample size. Furthermore, we remark that $R^3M$ is versatile and can be extended to various preference optimization methods, including direct preference optimization (DPO). Our experiments on robotic control and natural language generation with large language models (LLMs) show that $R^3M$ improves robustness of the reward against several types of perturbations to the preference data.