Doubly Robust Alignment for Large Language Models

TL;DR

This paper introduces a doubly robust preference optimization method for reinforcement learning from human feedback, improving robustness and consistency in aligning large language models with human preferences, even under model misspecification.

Contribution

It proposes a novel doubly robust algorithm that remains effective when either the preference model or reference policy is correctly specified, enhancing robustness over existing methods.

Findings

Outperforms state-of-the-art algorithms in robustness and accuracy.

Theoretically guarantees performance under model misspecification.

Demonstrates superior practical results in aligning language models.

Abstract

This paper studies reinforcement learning from human feedback (RLHF) for aligning large language models with human preferences. While RLHF has demonstrated promising results, many algorithms are highly sensitive to misspecifications in the underlying preference model (e.g., the Bradley-Terry model), the reference policy, or the reward function, resulting in undesirable fine-tuning. To address model misspecification, we propose a doubly robust preference optimization algorithm that remains consistent when either the preference model or the reference policy is correctly specified (without requiring both). Our proposal demonstrates superior and more robust performance than state-of-the-art algorithms, both in theory and in practice. The code is available at https://github.com/DRPO4LLM/DRPO4LLM