Difficulty-Based Preference Data Selection by DPO Implicit Reward Gap

TL;DR

This paper proposes a novel difficulty-based data selection method for preference datasets in LLM alignment, improving efficiency and performance by focusing on challenging examples with smaller DPO implicit reward gaps.

Contribution

It introduces a new data selection strategy based on DPO implicit reward gaps, enhancing data efficiency and model alignment in LLMs.

Findings

Outperforms five strong baselines across multiple datasets

Achieves superior performance with only 10% of original data

Improves data efficiency and model alignment

Abstract

Aligning large language models (LLMs) with human preferences is a critical challenge in AI research. While methods like Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO) are widely used, they often rely on large, costly preference datasets. The current work lacks methods for high-quality data selection specifically for preference data. In this work, we introduce a novel difficulty-based data selection strategy for preference datasets, grounded in the DPO implicit reward mechanism. By selecting preference data examples with smaller DPO implicit reward gaps, which are indicative of more challenging cases, we improve data efficiency and model alignment. Our approach consistently outperforms five strong baselines across multiple datasets and alignment tasks, achieving superior performance with only 10\% of the original data. This principled, efficient selection method offers a promising solution for scaling LLM alignment with limited resources.