DreamBoothDPO: Improving Personalized Generation using Direct Preference Optimization

TL;DR

This paper introduces DreamBoothDPO, an RL-based method that enhances personalized text-to-image generation by optimizing concept fidelity and contextual alignment without human annotations, using synthetic data and flexible trade-offs.

Contribution

It proposes a novel RL-based training approach that leverages synthetic quality metrics for improved personalized diffusion models, eliminating the need for human-annotated scores.

Findings

Outperforms naive baselines in convergence speed and quality

Effectively balances concept fidelity and prompt adherence

Demonstrates robustness across various architectures

Abstract

Personalized diffusion models have shown remarkable success in Text-to-Image (T2I) generation by enabling the injection of user-defined concepts into diverse contexts. However, balancing concept fidelity with contextual alignment remains a challenging open problem. In this work, we propose an RL-based approach that leverages the diverse outputs of T2I models to address this issue. Our method eliminates the need for human-annotated scores by generating a synthetic paired dataset for DPO-like training using external quality metrics. These better-worse pairs are specifically constructed to improve both concept fidelity and prompt adherence. Moreover, our approach supports flexible adjustment of the trade-off between image fidelity and textual alignment. Through multi-step training, our approach outperforms a naive baseline in convergence speed and output quality. We conduct extensive qualitative and quantitative analysis, demonstrating the effectiveness of our method across various architectures and fine-tuning techniques. The source code can be found at https://github.com/ControlGenAI/DreamBoothDPO.