AMaPO: Adaptive Margin-attached Preference Optimization for Language Model Alignment

TL;DR

AMaPO introduces an adaptive margin technique for offline preference optimization, improving language model alignment by dynamically balancing learning signals for correctly and incorrectly ranked samples, thus addressing overfitting and underfitting issues.

Contribution

The paper presents AMaPO, a novel algorithm that uses an adaptive margin with normalization and scaling to enhance preference optimization for language model alignment.

Findings

Achieves better ranking accuracy on benchmark datasets.

Improves downstream alignment performance.

Effectively mitigates overfitting and underfitting in preference optimization.

Abstract

Offline preference optimization offers a simpler and more stable alternative to RLHF for aligning language models. However, their effectiveness is critically dependent on ranking accuracy, a metric where further gains are highly impactful. This limitation arises from a fundamental problem that we identify and formalize as the Overfitting-Underfitting Dilemma: current margin designs cause models to apply excessive, wasteful gradients to correctly ranked samples (overfitting) while providing insufficient corrective signals for misranked ones (underfitting). To resolve this dilemma, we propose Adaptive Margin-attached Preference Optimization (AMaPO), a simple yet principled algorithm. AMaPO employs an instance-wise adaptive margin, refined by Z-normalization and exponential scaling, which dynamically reallocates learning effort by amplifying gradients for misranked samples and suppressing them for correct ones. Extensive experiments on widely used benchmarks demonstrate that AMaPO not only achieves better ranking accuracy and superior downstream alignment performance, but targeted analysis also confirms that it successfully mitigates the core overfitting and underfitting issues.