Preference Optimization with Multi-Sample Comparisons

TL;DR

This paper introduces multi-sample comparison methods for post-training generative models, enhancing the optimization of diversity and bias by capturing group-wise characteristics, and demonstrating robustness against label noise.

Contribution

It proposes novel multi-sample preference optimization techniques, mDPO and mIPO, extending existing preference methods to better optimize collective generative qualities.

Findings

Multi-sample comparison improves diversity and bias optimization.

Multi-sample methods are more robust to label noise.

Enhanced collective characteristic optimization over single-sample approaches.

Abstract

Recent advancements in generative models, particularly large language models (LLMs) and diffusion models, have been driven by extensive pretraining on large datasets followed by post-training. However, current post-training methods such as reinforcement learning from human feedback (RLHF) and direct alignment from preference methods (DAP) primarily utilize single-sample comparisons. These approaches often fail to capture critical characteristics such as generative diversity and bias, which are more accurately assessed through multiple samples. To address these limitations, we introduce a novel approach that extends post-training to include multi-sample comparisons. To achieve this, we propose Multi-sample Direct Preference Optimization (mDPO) and Multi-sample Identity Preference Optimization (mIPO). These methods improve traditional DAP methods by focusing on group-wise characteristics. Empirically, we demonstrate that multi-sample comparison is more effective in optimizing collective characteristics~(e.g., diversity and bias) for generative models than single-sample comparison. Additionally, our findings suggest that multi-sample comparisons provide a more robust optimization framework, particularly for dataset with label noise.