CAPO: Confidence Aware Preference Optimization Learning for Multilingual Preferences

TL;DR

CAPO introduces a confidence-aware method for preference optimization that improves multilingual model alignment and robustness over existing techniques like DPO, especially in noisy or low-margin scenarios.

Contribution

The paper presents CAPO, a novel preference optimization approach that dynamically scales loss based on confidence, enhancing multilingual robustness and outperforming previous methods.

Findings

CAPO outperforms existing baselines by at least 16% in reward accuracy.

CAPO improves alignment by increasing the margin between preferred and dispreferred responses.

CAPO demonstrates robustness to noisy and low-margin preference data.

Abstract

Preference optimization is a critical post-training technique used to align large language models (LLMs) with human preferences, typically by fine-tuning on ranked response pairs. While methods like Direct Preference Optimization (DPO) have proven effective in English, they often fail to generalize robustly to multilingual settings. We propose a simple yet effective alternative, Confidence-Aware Preference Optimization (CAPO), which replaces DPO's fixed treatment of preference pairs with a dynamic loss scaling mechanism based on a relative reward. By modulating the learning signal according to the confidence in each preference pair, CAPO enhances robustness to noisy or low-margin comparisons, typically encountered in multilingual text. Empirically, CAPO outperforms existing preference optimization baselines by at least 16% in reward accuracy, and improves alignment by widening the gap between preferred and dispreferred responses across languages.