DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulation

TL;DR

DiffusionCLIP introduces a text-guided image manipulation method using diffusion models, overcoming limitations of GAN-based approaches in handling diverse, real-world images with novel poses and contents, achieving robust zero-shot manipulation.

Contribution

The paper presents DiffusionCLIP, a novel diffusion model-based approach for zero-shot, text-guided image manipulation that handles diverse real images and multi-attribute editing.

Findings

Outperforms existing baselines in manipulation quality.

Effective manipulation across unseen domains and diverse datasets.

Human evaluations confirm robustness and superiority.

Abstract

Recently, GAN inversion methods combined with Contrastive Language-Image Pretraining (CLIP) enables zero-shot image manipulation guided by text prompts. However, their applications to diverse real images are still difficult due to the limited GAN inversion capability. Specifically, these approaches often have difficulties in reconstructing images with novel poses, views, and highly variable contents compared to the training data, altering object identity, or producing unwanted image artifacts. To mitigate these problems and enable faithful manipulation of real images, we propose a novel method, dubbed DiffusionCLIP, that performs text-driven image manipulation using diffusion models. Based on full inversion capability and high-quality image generation power of recent diffusion models, our method performs zero-shot image manipulation successfully even between unseen domains and takes another step towards general application by manipulating images from a widely varying ImageNet dataset. Furthermore, we propose a novel noise combination method that allows straightforward multi-attribute manipulation. Extensive experiments and human evaluation confirmed robust and superior manipulation performance of our methods compared to the existing baselines. Code is available at https://github.com/gwang-kim/DiffusionCLIP.git.