Clear Preferences Leave Traces: Reference Model-Guided Sampling for Preference Learning

TL;DR

This paper introduces a reference model-guided sampling method that improves preference learning efficiency and effectiveness by identifying high-quality training samples through reference model probabilities, reducing data needs and enhancing performance.

Contribution

The paper proposes a novel sampling strategy leveraging reference model probabilities to select high-quality samples, boosting preference learning outcomes with less data and resources.

Findings

Improves MT-Bench performance by 0.1 to 0.4 points with less than half the data

Achieves 0.4 to 0.98 point improvements on coding, math, and reasoning tasks

Reduces data requirements while maintaining or enhancing model alignment quality

Abstract

Direct Preference Optimization (DPO) has emerged as a de-facto approach for aligning language models with human preferences. Recent work has shown DPO's effectiveness relies on training data quality. In particular, clear quality differences between preferred and rejected responses enhance learning performance. Current methods for identifying and obtaining such high-quality samples demand additional resources or external models. We discover that reference model probability space naturally detects high-quality training samples. Using this insight, we present a sampling strategy that achieves consistent improvements (+0.1 to +0.4) on MT-Bench while using less than half (30-50%) of the training data. We observe substantial improvements (+0.4 to +0.98) for technical tasks (coding, math, and reasoning) across multiple models and hyperparameter settings.