Ambiguity Awareness Optimization: Towards Semantic Disambiguation for Direct Preference Optimization

TL;DR

This paper introduces Ambiguity Awareness Optimization (AAO), a novel method that re-weights ambiguous content during DPO training to improve alignment and performance in reinforcement learning from human feedback.

Contribution

The paper proposes AAO, a new approach that automatically reduces ambiguities in preference data, leading to significant performance gains over existing methods.

Findings

AAO outperforms DPO by up to 8.9 points on AlpacaEval 2.

AAO achieves up to 15.0 points improvement on Arena-Hard.

AAO is effective across multiple model scales and benchmark datasets.

Abstract

Direct Preference Optimization (DPO) is a widely used reinforcement learning from human feedback (RLHF) method across various domains. Recent research has increasingly focused on the role of token importance in improving DPO effectiveness. It is observed that identical or semantically similar content (defined as ambiguous content) frequently appears within the preference pairs. We hypothesize that the presence of ambiguous content during DPO training may introduce ambiguity, thereby limiting further improvements in alignment. Through mathematical analysis and proof-of-concept experiments, we reveal that ambiguous content may potentially introduce ambiguities, thereby degrading performance. To address this issue, we introduce Ambiguity Awareness Optimization (AAO), a simple yet effective approach that automatically re-weights ambiguous content to reduce ambiguities by calculating semantic similarity from preference pairs. Through extensive experiments, we demonstrate that AAO consistently and significantly surpasses state-of-the-art approaches in performance, without markedly increasing response length, across multiple model scales and widely adopted benchmark datasets, including AlpacaEval 2, MT-Bench, and Arena-Hard. Specifically, AAO outperforms DPO by up to 8.9 points on AlpacaEval 2 and achieves an improvement of by up to 15.0 points on Arena-Hard.