Self-Improving Robust Preference Optimization

TL;DR

This paper introduces SRPO, a novel offline RLHF framework that enables models to self-improve from human preferences, improving alignment and robustness across tasks without task-specific tuning.

Contribution

SRPO formulates preference learning as a task-independent min-max optimization, reformulating it into a scalable, non-adversarial offline loss for effective self-improvement.

Findings

SRPO outperforms DPO by 15% in AI Win-Rate on XSum after 5 revisions.

SRPO achieves 56% Win-Rate on Arena-Hard prompts, surpassing DPO and IPO.

The method is robust and task-independent, enabling effective self-improvement in preference optimization.

Abstract

Online and offline RLHF methods, such as PPO and DPO, have been highly successful in aligning AI with human preferences. Despite their success, however, these methods suffer from fundamental limitations: (a) Models trained with RLHF can learn from mistakes or negative examples through RL mechanism or contrastive loss during training. However, at inference time, they lack an innate self-improvement mechanism for error corrections. (b) The optimal solution of existing methods is highly task-dependent, making it difficult for them to generalize to new tasks. To address these challenges, we propose Self-Improving Robust Preference Optimization (SRPO), a practical and mathematically principled offline RLHF framework. The key idea behind SRPO is to cast the problem of learning from human preferences as a self-improvement process, mathematically formulated as a min-max objective that jointly optimizes a self-improvement policy and a generative policy in an adversarial fashion. Crucially, the solution for this optimization problem is independent of the training task, which makes it robust to its changes. We then show that this objective can be reformulated as a non-adversarial offline loss, which can be efficiently optimized using standard supervised learning techniques at scale. To demonstrate SRPO's effectiveness, we evaluate it using AI Win-Rate (WR) against human (GOLD) completions. When tested on the XSum dataset, SRPO outperforms DPO by a margin of 15% after 5 self revisions, achieving an impressive 90% WR. Moreover, on the challenging Arena-Hard prompts, SRPO outperforms both DPO and IPO (by 4% without revision and 6% after a single revision), reaching a 56% WR against against Llama-3.1-8B-Instruct.