TF-ICON: Diffusion-Based Training-Free Cross-Domain Image Composition

TL;DR

TF-ICON is a training-free framework that uses text-driven diffusion models for cross-domain image composition, enabling seamless object integration without additional training or fine-tuning.

Contribution

It introduces a novel training-free approach leveraging off-the-shelf diffusion models and an exceptional prompt for real image inversion, improving cross-domain image composition.

Findings

Outperforms state-of-the-art inversion methods on multiple datasets

Surpasses prior baselines in diverse visual domains

Operates without additional training or fine-tuning

Abstract

Text-driven diffusion models have exhibited impressive generative capabilities, enabling various image editing tasks. In this paper, we propose TF-ICON, a novel Training-Free Image COmpositioN framework that harnesses the power of text-driven diffusion models for cross-domain image-guided composition. This task aims to seamlessly integrate user-provided objects into a specific visual context. Current diffusion-based methods often involve costly instance-based optimization or finetuning of pretrained models on customized datasets, which can potentially undermine their rich prior. In contrast, TF-ICON can leverage off-the-shelf diffusion models to perform cross-domain image-guided composition without requiring additional training, finetuning, or optimization. Moreover, we introduce the exceptional prompt, which contains no information, to facilitate text-driven diffusion models in accurately inverting real images into latent representations, forming the basis for compositing. Our experiments show that equipping Stable Diffusion with the exceptional prompt outperforms state-of-the-art inversion methods on various datasets (CelebA-HQ, COCO, and ImageNet), and that TF-ICON surpasses prior baselines in versatile visual domains. Code is available at https://github.com/Shilin-LU/TF-ICON