Adaptive Batch-Wise Sample Scheduling for Direct Preference Optimization

TL;DR

This paper introduces SamS, an adaptive sample scheduling algorithm for DPO that dynamically selects training samples based on the model's evolving state, significantly enhancing LLM alignment performance.

Contribution

The paper proposes a novel adaptive sample scheduling method, SamS, for DPO that improves training efficiency and model alignment without altering the core algorithm.

Findings

SamS improves performance across multiple tasks.

Integration of SamS requires minimal additional computational cost.

Sample scheduling based on model feedback enhances generalization.

Abstract

Direct Preference Optimization (DPO) has emerged as an effective approach for aligning large language models (LLMs) with human preferences. However, its performance is highly dependent on the quality of the underlying human preference data. To address this bottleneck, prior work has explored various data selection strategies, but these methods often overlook the impact of the evolving states of the language model during the optimization process. In this paper, we introduce a novel problem: Sample Scheduling for DPO, which aims to dynamically and adaptively schedule training samples based on the model's evolving batch-wise states throughout preference optimization. To solve this problem, we propose SamS, an efficient and effective algorithm that adaptively selects samples in each training batch based on the LLM's learning feedback to maximize the potential generalization performance. Notably, without modifying the core DPO algorithm, simply integrating SamS significantly improves performance across tasks, with minimal additional computational overhead. This work points to a promising new direction for improving LLM alignment through batch-wise sample selection, with potential generalization to RLHF and broader supervised learning paradigms.