GroupDPO: Memory efficient Group-wise Direct Preference Optimization

TL;DR

This paper introduces GroupDPO, a memory-efficient algorithm for group-wise preference optimization that improves LLM alignment by utilizing multiple responses, reducing memory usage, and enhancing training stability.

Contribution

It proposes a novel memory-efficient optimization method that enables scalable group-wise preference training, outperforming single-pair approaches.

Findings

GroupDPO reduces peak memory usage during training.

Leveraging multiple responses improves alignment performance.

Adding NLL on positive responses enhances stability and results.

Abstract

Preference optimization is widely used to align Large Language Models (LLMs) with preference feedback. However, most existing methods train on a single positive-negative pair per prompt, discarding additional supervision available in preference datasets that typically contain multiple candidate responses. Motivated by this limitation, recent work explores group-wise preference optimization, which jointly contrasts multiple responses for the same prompt, but its empirical behavior and scalability remain underexplored due to the memory overhead of group-coupled objectives. In this work, we introduce a memory-efficient group-wise preference optimization algorithm that preserves gradients while decoupling samples during backpropagation, substantially reducing peak memory usage, which enables scalable training with larger group sizes. Across both offline and online alignment settings, we show that leveraging multiple responses consistently outperforms single-pair training. Furthermore, incorporating a negative log-likelihood (NLL) term on positive responses is critical for both performance gains and training stability.