Test-time Conditional Text-to-Image Synthesis Using Diffusion Models

TL;DR

This paper introduces TINTIN, a training-free, test-time method for conditional text-to-image synthesis with diffusion models, enabling flexible control over outputs using various conditioning factors like color and edges.

Contribution

The paper presents TINTIN, a novel test-time algorithm that manipulates diffusion model outputs without retraining, allowing control with multiple conditioning inputs such as color palettes and edge maps.

Findings

Significant qualitative improvements over state-of-the-art methods.

Effective control of generated images using color palettes and edge maps.

Demonstrated flexibility and extensibility of the approach.

Abstract

We consider the problem of conditional text-to-image synthesis with diffusion models. Most recent works need to either finetune specific parts of the base diffusion model or introduce new trainable parameters, leading to deployment inflexibility due to the need for training. To address this gap in the current literature, we propose our method called TINTIN: Test-time Conditional Text-to-Image Synthesis using Diffusion Models which is a new training-free test-time only algorithm to condition text-to-image diffusion model outputs on conditioning factors such as color palettes and edge maps. In particular, we propose to interpret noise predictions during denoising as gradients of an energy-based model, leading to a flexible approach to manipulate the noise by matching predictions inferred from them to the ground truth conditioning input. This results in, to the best of our knowledge, the first approach to control model outputs with input color palettes, which we realize using a novel color distribution matching loss. We also show this test-time noise manipulation can be easily extensible to other types of conditioning, e.g., edge maps. We conduct extensive experiments using a variety of text prompts, color palettes, and edge maps and demonstrate significant improvement over the current state-of-the-art, both qualitatively and quantitatively.