TKG-DM: Training-free Chroma Key Content Generation Diffusion Model

TL;DR

This paper introduces TKG-DM, a training-free diffusion model that enables precise control over foreground and background separation in image generation, especially for chroma key backgrounds, without the need for fine-tuning.

Contribution

It is the first method to manipulate initial noise color for controlled background generation in diffusion models without training, outperforming fine-tuned models in quality.

Findings

Outperforms existing methods in qualitative evaluations.

Matches or surpasses fine-tuned models in quantitative metrics.

Extends to other tasks like text-to-video and consistency models.

Abstract

Diffusion models have enabled the generation of high-quality images with a strong focus on realism and textual fidelity. Yet, large-scale text-to-image models, such as Stable Diffusion, struggle to generate images where foreground objects are placed over a chroma key background, limiting their ability to separate foreground and background elements without fine-tuning. To address this limitation, we present a novel Training-Free Chroma Key Content Generation Diffusion Model (TKG-DM), which optimizes the initial random noise to produce images with foreground objects on a specifiable color background. Our proposed method is the first to explore the manipulation of the color aspects in initial noise for controlled background generation, enabling precise separation of foreground and background without fine-tuning. Extensive experiments demonstrate that our training-free method outperforms existing methods in both qualitative and quantitative evaluations, matching or surpassing fine-tuned models. Finally, we successfully extend it to other tasks (e.g., consistency models and text-to-video), highlighting its transformative potential across various generative applications where independent control of foreground and background is crucial.