When Data is the Algorithm: A Systematic Study and Curation of Preference Optimization Datasets

TL;DR

This paper conducts a comprehensive data-centric analysis of open-source preference datasets for LLM alignment, introduces a new curated dataset UltraMix, and provides insights to improve preference-based fine-tuning.

Contribution

It systematically annotates and compares existing DPO datasets using the Magpie framework and creates UltraMix, a refined dataset that enhances performance while reducing size.

Findings

UltraMix outperforms individual datasets on key benchmarks.

Annotations reveal structural and qualitative differences in datasets.

UltraMix is 30% smaller yet more effective than the best single dataset.

Abstract

Aligning large language models (LLMs) is a central objective of post-training, often achieved through reward modeling and reinforcement learning methods. Among these, direct preference optimization (DPO) has emerged as a widely adopted technique that fine-tunes LLMs on preferred completions over less favorable ones. While most frontier LLMs do not disclose their curated preference pairs, the broader LLM community has released several open-source DPO datasets, including TuluDPO, ORPO, UltraFeedback, HelpSteer, and Code-Preference-Pairs. However, systematic comparisons remain scarce, largely due to the high computational cost and the lack of rich quality annotations, making it difficult to understand how preferences were selected, which task types they span, and how well they reflect human judgment on a per-sample level. In this work, we present the first comprehensive, data-centric analysis of popular open-source DPO corpora. We leverage the Magpie framework to annotate each sample for task category, input quality, and preference reward, a reward-model-based signal that validates the preference order without relying on human annotations. This enables a scalable, fine-grained inspection of preference quality across datasets, revealing structural and qualitative discrepancies in reward margins. Building on these insights, we systematically curate a new DPO mixture, UltraMix, that draws selectively from all five corpora while removing noisy or redundant samples. UltraMix is 30% smaller than the best-performing individual dataset yet exceeds its performance across key benchmarks. We publicly release all annotations, metadata, and our curated mixture to facilitate future research in data-centric preference optimization.