ControlNet++: Improving Conditional Controls with Efficient Consistency Feedback

TL;DR

ControlNet++ enhances the controllability of text-to-image diffusion models by explicitly optimizing pixel-level cycle consistency with an efficient reward strategy, leading to significant improvements in various conditional controls.

Contribution

It introduces a novel cycle consistency optimization method with an efficient reward strategy to improve controllable image generation in diffusion models.

Findings

Achieves 11.1% improvement in mIoU for segmentation masks

Achieves 13.4% improvement in SSIM for line-art edges

Achieves 7.6% reduction in RMSE for depth conditions

Abstract

To enhance the controllability of text-to-image diffusion models, existing efforts like ControlNet incorporated image-based conditional controls. In this paper, we reveal that existing methods still face significant challenges in generating images that align with the image conditional controls. To this end, we propose ControlNet++, a novel approach that improves controllable generation by explicitly optimizing pixel-level cycle consistency between generated images and conditional controls. Specifically, for an input conditional control, we use a pre-trained discriminative reward model to extract the corresponding condition of the generated images, and then optimize the consistency loss between the input conditional control and extracted condition. A straightforward implementation would be generating images from random noises and then calculating the consistency loss, but such an approach requires storing gradients for multiple sampling timesteps, leading to considerable time and memory costs. To address this, we introduce an efficient reward strategy that deliberately disturbs the input images by adding noise, and then uses the single-step denoised images for reward fine-tuning. This avoids the extensive costs associated with image sampling, allowing for more efficient reward fine-tuning. Extensive experiments show that ControlNet++ significantly improves controllability under various conditional controls. For example, it achieves improvements over ControlNet by 11.1% mIoU, 13.4% SSIM, and 7.6% RMSE, respectively, for segmentation mask, line-art edge, and depth conditions. All the code, models, demo and organized data have been open sourced on our Github Repo.