Calibrated Multi-Preference Optimization for Aligning Diffusion Models

TL;DR

This paper introduces CaPO, a novel method for aligning diffusion models with multiple preferences using reward calibration and Pareto frontier pair selection, eliminating the need for human annotations.

Contribution

The paper proposes Calibrated Preference Optimization (CaPO), a new approach that leverages multiple reward models and calibration techniques to improve preference alignment without human data.

Findings

CaPO outperforms prior methods like DPO on T2I benchmarks.

It effectively handles multi-preference scenarios.

The method improves alignment accuracy and generalization.

Abstract

Aligning text-to-image (T2I) diffusion models with preference optimization is valuable for human-annotated datasets, but the heavy cost of manual data collection limits scalability. Using reward models offers an alternative, however, current preference optimization methods fall short in exploiting the rich information, as they only consider pairwise preference distribution. Furthermore, they lack generalization to multi-preference scenarios and struggle to handle inconsistencies between rewards. To address this, we present Calibrated Preference Optimization (CaPO), a novel method to align T2I diffusion models by incorporating the general preference from multiple reward models without human annotated data. The core of our approach involves a reward calibration method to approximate the general preference by computing the expected win-rate against the samples generated by the pretrained models. Additionally, we propose a frontier-based pair selection method that effectively manages the multi-preference distribution by selecting pairs from Pareto frontiers. Finally, we use regression loss to fine-tune diffusion models to match the difference between calibrated rewards of a selected pair. Experimental results show that CaPO consistently outperforms prior methods, such as Direct Preference Optimization (DPO), in both single and multi-reward settings validated by evaluation on T2I benchmarks, including GenEval and T2I-Compbench.