Mod-Adapter: Tuning-Free and Versatile Multi-concept Personalization via Modulation Adapter

TL;DR

This paper introduces Mod-Adapter, a tuning-free method for multi-concept personalization in text-to-image generation, capable of customizing both object and abstract concepts without fine-tuning, using a novel modulation mechanism and vision-language guidance.

Contribution

The paper presents Mod-Adapter, a novel tuning-free approach that leverages a modulation mechanism and vision-language pre-training to personalize multiple concepts, including abstract ones, without test-time fine-tuning.

Findings

Achieves state-of-the-art results in multi-concept personalization.

Supports both object and abstract concept customization.

Outperforms existing methods in quantitative, qualitative, and human evaluations.

Abstract

Personalized text-to-image generation aims to synthesize images of user-provided concepts in diverse contexts. Despite recent progress in multi-concept personalization, most are limited to object concepts and struggle to customize abstract concepts (e.g., pose, lighting). Some methods have begun exploring multi-concept personalization supporting abstract concepts, but they require test-time fine-tuning for each new concept, which is time-consuming and prone to overfitting on limited training images. In this work, we propose a novel tuning-free method for multi-concept personalization that can effectively customize both object and abstract concepts without test-time fine-tuning. Our method builds upon the modulation mechanism in pre-trained Diffusion Transformers (DiTs) model, leveraging the localized and semantically meaningful properties of the modulation space. Specifically, we propose a novel module, Mod-Adapter, to predict concept-specific modulation direction for the modulation process of concept-related text tokens. It introduces vision-language cross-attention for extracting concept visual features, and Mixture-of-Experts (MoE) layers that adaptively map the concept features into the modulation space. Furthermore, to mitigate the training difficulty caused by the large gap between the concept image space and the modulation space, we introduce a VLM-guided pre-training strategy that leverages the strong image understanding capabilities of vision-language models to provide semantic supervision signals. For a comprehensive comparison, we extend a standard benchmark by incorporating abstract concepts. Our method achieves state-of-the-art performance in multi-concept personalization, supported by quantitative, qualitative, and human evaluations.