Clone-Robust AI Alignment

TL;DR

This paper addresses the challenge of aligning large language models with human preferences by introducing a robust RLHF algorithm that maintains performance even with unbalanced datasets, inspired by social choice theory.

Contribution

The paper proposes a novel weighted MLE algorithm for RLHF that ensures robustness to approximate clones, improving alignment reliability in unbalanced datasets.

Findings

Standard RLHF fails to be clone-robust.

Weighted MLE guarantees robustness to approximate clones.

The new method preserves theoretical properties of RLHF.

Abstract

A key challenge in training Large Language Models (LLMs) is properly aligning them with human preferences. Reinforcement Learning with Human Feedback (RLHF) uses pairwise comparisons from human annotators to train reward functions and has emerged as a popular alignment method. However, input datasets in RLHF are not necessarily balanced in the types of questions and answers that are included. Therefore, we want RLHF algorithms to perform well even when the set of alternatives is not uniformly distributed. Drawing on insights from social choice theory, we introduce robustness to approximate clones, a desirable property of RLHF algorithms which requires that adding near-duplicate alternatives does not significantly change the learned reward function. We first demonstrate that the standard RLHF algorithm based on regularized maximum likelihood estimation (MLE) fails to satisfy this property. We then propose the weighted MLE, a new RLHF algorithm that modifies the standard regularized MLE by weighting alternatives based on their similarity to other alternatives. This new algorithm guarantees robustness to approximate clones while preserving desirable theoretical properties.