Adding Conditional Control to Text-to-Image Diffusion Models

TL;DR

ControlNet is a neural network architecture that adds spatial conditioning controls to large pretrained text-to-image diffusion models, enabling diverse and robust image generation with various conditions.

Contribution

It introduces ControlNet, a novel architecture that reuses pretrained diffusion models and adds controllable spatial conditioning with zero-initialized layers for flexible image synthesis.

Findings

ControlNet effectively incorporates multiple spatial controls like edges, depth, and pose.

Training is robust across small and large datasets.

ControlNet enhances the versatility of diffusion models for controlled image generation.

Abstract

We present ControlNet, a neural network architecture to add spatial conditioning controls to large, pretrained text-to-image diffusion models. ControlNet locks the production-ready large diffusion models, and reuses their deep and robust encoding layers pretrained with billions of images as a strong backbone to learn a diverse set of conditional controls. The neural architecture is connected with "zero convolutions" (zero-initialized convolution layers) that progressively grow the parameters from zero and ensure that no harmful noise could affect the finetuning. We test various conditioning controls, eg, edges, depth, segmentation, human pose, etc, with Stable Diffusion, using single or multiple conditions, with or without prompts. We show that the training of ControlNets is robust with small (<50k) and large (>1m) datasets. Extensive results show that ControlNet may facilitate wider applications to control image diffusion models.