MASS-DPO: Multi-negative Active Sample Selection for Direct Policy Optimization

TL;DR

MASS-DPO introduces an active sampling method for selecting informative negative samples in preference optimization, improving efficiency and accuracy across multiple benchmarks.

Contribution

It develops a Fisher-information-based objective for selecting compact, diverse negative samples, reducing redundancy and enhancing policy training.

Findings

Outperforms existing methods in accuracy across four benchmarks.

Achieves better recall, NDCG, and optimization dynamics.

Requires fewer negative samples to reach comparable or better performance.

Abstract

Multi-negative preference optimization under the Plackett--Luce (PL) model extends Direct Preference Optimization (DPO) by leveraging comparative signals across one preferred and multiple rejected responses. However, optimizing over large negative pools is costly, and many candidates contribute redundant gradients due to their similar effects on policy updates. We introduce MASS-DPO, a multi-negative active sample selection method that derives a PL-specific Fisher-information objective for selecting compact, informative negative subsets within each prompt. The resulting log-determinant objective selects negatives that contribute complementary information for policy updates, yielding compact subsets that retain the full pool's information while reducing redundancy. In practice, this favors negatives whose gradients cover different update directions, reducing redundant signal from near-duplicate candidates while preserving the most useful training information. Across four benchmarks spanning recommendation and multiple-choice QA and three model families, MASS-DPO consistently exceeds or matches existing methods in accuracy, improves Recall/NDCG and margin-based optimization dynamics, and delivers stronger alignment with substantially fewer negatives.