Resolving Multi-Condition Confusion for Finetuning-Free Personalized Image Generation

TL;DR

This paper introduces a weighted-merge method to resolve object confusion in finetuning-free personalized image generation with multiple references, improving accuracy and reducing training costs.

Contribution

It proposes a novel weighted-merge technique for multi-object reference integration and an object quality score for training sample selection, enhancing personalization without finetuning.

Findings

Outperforms state-of-the-art on Concept101 and DreamBooth datasets.

Effectively reduces object confusion in multi-object generation.

Improves single-object generation with multiple references.

Abstract

Personalized text-to-image generation methods can generate customized images based on the reference images, which have garnered wide research interest. Recent methods propose a finetuning-free approach with a decoupled cross-attention mechanism to generate personalized images requiring no test-time finetuning. However, when multiple reference images are provided, the current decoupled cross-attention mechanism encounters the object confusion problem and fails to map each reference image to its corresponding object, thereby seriously limiting its scope of application. To address the object confusion problem, in this work we investigate the relevance of different positions of the latent image features to the target object in diffusion model, and accordingly propose a weighted-merge method to merge multiple reference image features into the corresponding objects. Next, we integrate this weighted-merge method into existing pre-trained models and continue to train the model on a multi-object dataset constructed from the open-sourced SA-1B dataset. To mitigate object confusion and reduce training costs, we propose an object quality score to estimate the image quality for the selection of high-quality training samples. Furthermore, our weighted-merge training framework can be employed on single-object generation when a single object has multiple reference images. The experiments verify that our method achieves superior performance to the state-of-the-arts on the Concept101 dataset and DreamBooth dataset of multi-object personalized image generation, and remarkably improves the performance on single-object personalized image generation. Our code is available at https://github.com/hqhQAQ/MIP-Adapter.