Plug-and-Play Multi-Concept Adaptive Blending for High-Fidelity Text-to-Image Synthesis

TL;DR

This paper introduces PnP-MIX, a tuning-free method for high-fidelity text-to-image synthesis that effectively integrates multiple personalized concepts with improved fidelity, localization, and without model tuning.

Contribution

The paper presents a novel plug-and-play adaptive blending approach that enhances multi-concept personalization in T2I generation, addressing limitations of existing methods.

Findings

Outperforms existing methods in multi-concept personalization

Maintains high fidelity and semantic consistency

Operates without additional model tuning

Abstract

Integrating multiple personalized concepts into a single image has recently become a significant area of focus within Text-to-Image (T2I) generation. However, existing methods often underperform on complex multi-object scenes due to unintended alterations in both personalized and non-personalized regions. This not only fails to preserve the intended prompt structure but also disrupts interactions among regions, leading to semantic inconsistencies. To address this limitation, we introduce plug-and-play multi-concept adaptive blending for high-fidelity text-to-image synthesis (PnP-MIX), an innovative, tuning-free approach designed to seamlessly embed multiple personalized concepts into a single generated image. Our method leverages guided appearance attention to faithfully reflect the intended appearance of each personalized concept. To further enhance compositional fidelity, we present a mask-guided noise mixing strategy that preserves the integrity of non-personalized regions such as the background or unrelated objects while enabling the precise integration of personalized objects. Finally, to mitigate concept leakage, i.e., the inadvertent leakage of personalized concept features into other regions, we propose background dilution++, a novel strategy that effectively reduces such leakage and promotes accurate localization of features within personalized regions. Extensive experimental results demonstrate that PnP-MIX consistently surpasses existing methodologies in both single- and multi-concept personalization scenarios, underscoring its robustness and superior performance without additional model tuning.