Unsupervised Region-Based Image Editing of Denoising Diffusion Models

TL;DR

This paper introduces a novel unsupervised method for semantic discovery and editing in the latent space of pre-trained diffusion models, enabling precise local image modifications without additional training or annotations.

Contribution

It proposes a Jacobian-based projection technique to identify semantic attributes in the latent space without supervision, surpassing supervised methods in some face attribute editing tasks.

Findings

Achieves state-of-the-art performance across multiple datasets.

Outperforms supervised approaches in specific face attribute editing.

Enables precise local image editing without annotations.

Abstract

Although diffusion models have achieved remarkable success in the field of image generation, their latent space remains under-explored. Current methods for identifying semantics within latent space often rely on external supervision, such as textual information and segmentation masks. In this paper, we propose a method to identify semantic attributes in the latent space of pre-trained diffusion models without any further training. By projecting the Jacobian of the targeted semantic region into a low-dimensional subspace which is orthogonal to the non-masked regions, our approach facilitates precise semantic discovery and control over local masked areas, eliminating the need for annotations. We conducted extensive experiments across multiple datasets and various architectures of diffusion models, achieving state-of-the-art performance. In particular, for some specific face attributes, the performance of our proposed method even surpasses that of supervised approaches, demonstrating its superior ability in editing local image properties.