Margin-aware Preference Optimization for Aligning Diffusion Models without Reference

TL;DR

This paper introduces MaPO, a reference-agnostic preference optimization method that improves alignment of diffusion models across diverse tasks by addressing the limitations of reference-based methods like DPO.

Contribution

MaPO is a novel, reference-free approach that directly optimizes preference likelihood margins, enabling more effective and versatile diffusion model alignment without reference mismatch issues.

Findings

MaPO outperforms DPO and DreamBooth across five domains.

MaPO reduces training time by 15%.

MaPO is more robust to reference mismatch severity.

Abstract

Modern preference alignment methods, such as DPO, rely on divergence regularization to a reference model for training stability-but this creates a fundamental problem we call "reference mismatch." In this paper, we investigate the negative impacts of reference mismatch in aligning text-to-image (T2I) diffusion models, showing that larger reference mismatch hinders effective adaptation given the same amount of data, e.g., as when learning new artistic styles, or personalizing to specific objects. We demonstrate this phenomenon across text-to-image (T2I) diffusion models and introduce margin-aware preference optimization (MaPO), a reference-agnostic approach that breaks free from this constraint. By directly optimizing the likelihood margin between preferred and dispreferred outputs under the Bradley-Terry model without anchoring to a reference, MaPO transforms diverse T2I tasks into unified pairwise preference optimization. We validate MaPO's versatility across five challenging domains: (1) safe generation, (2) style adaptation, (3) cultural representation, (4) personalization, and (5) general preference alignment. Our results reveal that MaPO's advantage grows dramatically with reference mismatch severity, outperforming both DPO and specialized methods like DreamBooth while reducing training time by 15%. MaPO thus emerges as a versatile and memory-efficient method for generic T2I adaptation tasks.