A Simple Approach to Unifying Diffusion-based Conditional Generation

TL;DR

This paper presents a simple, unified diffusion-based framework for diverse conditional image generation tasks, achieving comparable or better results than specialized or complex models with minimal additional parameters.

Contribution

The authors introduce a single, efficient diffusion model that unifies various conditional generation tasks without complex training or architectural modifications.

Findings

Comparable results to specialized methods

Better than prior unified approaches

Supports multi-signal conditional generation

Abstract

Recent progress in image generation has sparked research into controlling these models through condition signals, with various methods addressing specific challenges in conditional generation. Instead of proposing another specialized technique, we introduce a simple, unified framework to handle diverse conditional generation tasks involving a specific image-condition correlation. By learning a joint distribution over a correlated image pair (e.g. image and depth) with a diffusion model, our approach enables versatile capabilities via different inference-time sampling schemes, including controllable image generation (e.g. depth to image), estimation (e.g. image to depth), signal guidance, joint generation (image & depth), and coarse control. Previous attempts at unification often introduce significant complexity through multi-stage training, architectural modification, or increased parameter counts. In contrast, our simple formulation requires a single, computationally efficient training stage, maintains the standard model input, and adds minimal learned parameters (15% of the base model). Moreover, our model supports additional capabilities like non-spatially aligned and coarse conditioning. Extensive results show that our single model can produce comparable results with specialized methods and better results than prior unified methods. We also demonstrate that multiple models can be effectively combined for multi-signal conditional generation.