DCRM: A Heuristic to Measure Response Pair Quality in Preference Optimization

TL;DR

This paper introduces DCRM, a metric to evaluate response pair quality in preference optimization, and demonstrates that selecting pairs with high DCRM improves large language models' performance across multiple benchmarks.

Contribution

The paper proposes DCRM, a novel metric for assessing response pair quality, and a pairing method that enhances model training effectiveness in preference optimization.

Findings

Higher DCRM correlates with better learning outcomes.

Using DCRM-based pairing improves model performance on benchmarks.

DCRM effectively quantifies the quality of response pairs for preference learning.

Abstract

Recent research has attempted to associate preference optimization (PO) performance with the underlying preference datasets. In this work, our observation is that the differences between the preferred response $y^+$ and dispreferred response $y^-$ influence what LLMs can learn, which may not match the desirable differences to learn. Therefore, we use distance and reward margin to quantify these differences, and combine them to get Distance Calibrated Reward Margin (DCRM), a metric that measures the quality of a response pair for PO. Intuitively, DCRM encourages minimal noisy differences and maximal desired differences. With this, we study 3 types of commonly used preference datasets, classified along two axes: the source of the responses and the preference labeling function. We establish a general correlation between higher DCRM of the training set and better learning outcome. Inspired by this, we propose a best-of-$N^2$ pairing method that selects response pairs with the highest DCRM. Empirically, in various settings, our method produces training datasets that can further improve models' performance on AlpacaEval, MT-Bench, and Arena-Hard over the existing training sets.