Controllable Generation with Text-to-Image Diffusion Models: A Survey

TL;DR

This survey reviews recent advances in controllable text-to-image diffusion models, analyzing theoretical mechanisms and practical methods for conditionally guiding image generation beyond simple text prompts.

Contribution

It provides a comprehensive overview of controlling mechanisms in T2I diffusion models, categorizing approaches based on different conditioning strategies and offering insights into theoretical foundations and practical implementations.

Findings

Revealed how conditions are integrated into diffusion models.

Categorized controllable generation methods into specific, multiple, and universal conditions.

Provided a curated repository of related literature.

Abstract

In the rapidly advancing realm of visual generation, diffusion models have revolutionized the landscape, marking a significant shift in capabilities with their impressive text-guided generative functions. However, relying solely on text for conditioning these models does not fully cater to the varied and complex requirements of different applications and scenarios. Acknowledging this shortfall, a variety of studies aim to control pre-trained text-to-image (T2I) models to support novel conditions. In this survey, we undertake a thorough review of the literature on controllable generation with T2I diffusion models, covering both the theoretical foundations and practical advancements in this domain. Our review begins with a brief introduction to the basics of denoising diffusion probabilistic models (DDPMs) and widely used T2I diffusion models. We then reveal the controlling mechanisms of diffusion models, theoretically analyzing how novel conditions are introduced into the denoising process for conditional generation. Additionally, we offer a detailed overview of research in this area, organizing it into distinct categories from the condition perspective: generation with specific conditions, generation with multiple conditions, and universal controllable generation. For an exhaustive list of the controllable generation literature surveyed, please refer to our curated repository at https://github.com/PRIV-Creation/Awesome-Controllable-T2I-Diffusion-Models.