Customized Generation Reimagined: Fidelity and Editability Harmonized

TL;DR

This paper introduces a novel framework for customized text-to-image generation that balances concept fidelity and prompt editability by decoupling and selectively integrating two specialized branches, along with an image-specific context optimization strategy.

Contribution

The proposed Divide, Conquer, then Integrate (DCI) framework and ICO strategy enable simultaneous high fidelity and editability in customized generation models, overcoming previous trade-offs.

Findings

Effective reconciliation of fidelity and editability demonstrated.

Outperforms previous methods in concept fidelity and prompt adherence.

Adaptive image-specific fine-tuning improves overall performance.

Abstract

Customized generation aims to incorporate a novel concept into a pre-trained text-to-image model, enabling new generations of the concept in novel contexts guided by textual prompts. However, customized generation suffers from an inherent trade-off between concept fidelity and editability, i.e., between precisely modeling the concept and faithfully adhering to the prompts. Previous methods reluctantly seek a compromise and struggle to achieve both high concept fidelity and ideal prompt alignment simultaneously. In this paper, we propose a Divide, Conquer, then Integrate (DCI) framework, which performs a surgical adjustment in the early stage of denoising to liberate the fine-tuned model from the fidelity-editability trade-off at inference. The two conflicting components in the trade-off are decoupled and individually conquered by two collaborative branches, which are then selectively integrated to preserve high concept fidelity while achieving faithful prompt adherence. To obtain a better fine-tuned model, we introduce an Image-specific Context Optimization} (ICO) strategy for model customization. ICO replaces manual prompt templates with learnable image-specific contexts, providing an adaptive and precise fine-tuning direction to promote the overall performance. Extensive experiments demonstrate the effectiveness of our method in reconciling the fidelity-editability trade-off.