Personalized Preference Fine-tuning of Diffusion Models

TL;DR

This paper presents PPD, a multi-reward fine-tuning method for diffusion models that personalizes image generation to individual user preferences using few-shot learning and cross-attention with preference embeddings.

Contribution

Introduces PPD, a novel multi-reward optimization framework that personalizes diffusion models by learning individual preferences from limited examples and generalizes to unseen users.

Findings

Achieves 76% win rate in reflecting user preferences with four examples

Effectively optimizes for multiple reward functions simultaneously

Enables interpolation between different user preferences during inference

Abstract

RLHF techniques like DPO can significantly improve the generation quality of text-to-image diffusion models. However, these methods optimize for a single reward that aligns model generation with population-level preferences, neglecting the nuances of individual users' beliefs or values. This lack of personalization limits the efficacy of these models. To bridge this gap, we introduce PPD, a multi-reward optimization objective that aligns diffusion models with personalized preferences. With PPD, a diffusion model learns the individual preferences of a population of users in a few-shot way, enabling generalization to unseen users. Specifically, our approach (1) leverages a vision-language model (VLM) to extract personal preference embeddings from a small set of pairwise preference examples, and then (2) incorporates the embeddings into diffusion models through cross attention. Conditioning on user embeddings, the text-to-image models are fine-tuned with the DPO objective, simultaneously optimizing for alignment with the preferences of multiple users. Empirical results demonstrate that our method effectively optimizes for multiple reward functions and can interpolate between them during inference. In real-world user scenarios, with as few as four preference examples from a new user, our approach achieves an average win rate of 76\% over Stable Cascade, generating images that more accurately reflect specific user preferences.