Premier: Personalized Preference Modulation with Learnable User Embedding in Text-to-Image Generation

TL;DR

Premier introduces a personalized image generation framework that uses learnable user embeddings and a preference adapter to better capture and modulate individual user preferences in text-to-image synthesis.

Contribution

It proposes a novel preference modulation method with a dispersion loss for distinct user embeddings, improving personalization accuracy and generalization with limited data.

Findings

Premier outperforms prior methods in preference alignment.

It achieves higher scores on text consistency and ViPer proxy metrics.

Expert evaluations favor Premier's personalized outputs.

Abstract

Text-to-image generation has advanced rapidly, yet it still struggles to capture the nuanced user preferences. Existing approaches typically rely on multimodal large language models to infer user preferences, but the derived prompts or latent codes rarely reflect them faithfully, leading to suboptimal personalization. We present Premier, a novel preference modulation framework for personalized image generation. Premier represents each user's preference as a learnable embedding and introduces a preference adapter that fuses the user embedding with the text prompt. To enable accurate and fine-grained preference control, the fused preference embedding is further used to modulate the generative process. To enhance the distinctness of individual preference and improve alignment between outputs and user-specific styles, we incorporate a dispersion loss that enforces separation among user embeddings. When user data are scarce, new users are represented as linear combinations of existing preference embeddings learned during training, enabling effective generalization. Experiments show that Premier outperforms prior methods under the same history length, achieving stronger preference alignment and superior performance on text consistency, ViPer proxy metrics, and expert evaluations.