ActiveDPO: Active Direct Preference Optimization for Sample-Efficient Alignment

TL;DR

ActiveDPO introduces a theoretically grounded, LLM-aware active data selection method for preference-based alignment, significantly reducing data collection costs and improving alignment quality.

Contribution

It presents a novel active data selection algorithm that accounts for the LLM's influence, outperforming existing methods in sample-efficient alignment.

Findings

ActiveDPO outperforms existing methods across multiple models.

It effectively reduces the amount of human preference data needed.

The method demonstrates superior performance on real-world datasets.

Abstract

The recent success in using human preferences to align large language models (LLMs) has significantly improved their performance in various downstream tasks, such as question answering, mathematical reasoning, and code generation. However, achieving effective LLM alignment depends on high-quality datasets of human preferences. Collecting these datasets requires human preference annotation, which is costly and resource-intensive, necessitating efficient active data selection methods. Existing methods either lack a strong theoretical foundation or depend on restrictive assumptions about the reward function, such as linear latent reward functions. To this end, we propose an algorithm, ActiveDPO, that uses a theoretically grounded data selection criterion for non-linear reward functions while directly leveraging the LLM itself to parameterize the reward model used for active data selection. As a result, ActiveDPO explicitly accounts for the LLM's influence on data selection, unlike methods that select data without considering the LLM that is being aligned, thereby leading to more effective and efficient data collection. Our extensive experiments demonstrate that ActiveDPO outperforms existing methods across various models and real-world preference datasets.