Exploring Low-Dimensional Subspaces in Diffusion Models for Controllable Image Editing

TL;DR

This paper reveals that diffusion models have low-dimensional semantic subspaces within certain noise levels, enabling unsupervised, training-free, precise image editing through a novel method called LOCO Edit.

Contribution

It provides a theoretical understanding of the semantic structure in diffusion models and introduces LOCO Edit, a new method for controllable image editing without additional training.

Findings

LOCO Edit effectively identifies editing directions with desirable properties.

The method achieves precise local editing in diffusion models.

Empirical results demonstrate LOCO Edit's efficiency and effectiveness.

Abstract

Recently, diffusion models have emerged as a powerful class of generative models. Despite their success, there is still limited understanding of their semantic spaces. This makes it challenging to achieve precise and disentangled image generation without additional training, especially in an unsupervised way. In this work, we improve the understanding of their semantic spaces from intriguing observations: among a certain range of noise levels, (1) the learned posterior mean predictor (PMP) in the diffusion model is locally linear, and (2) the singular vectors of its Jacobian lie in low-dimensional semantic subspaces. We provide a solid theoretical basis to justify the linearity and low-rankness in the PMP. These insights allow us to propose an unsupervised, single-step, training-free LOw-rank COntrollable image editing (LOCO Edit) method for precise local editing in diffusion models. LOCO Edit identified editing directions with nice properties: homogeneity, transferability, composability, and linearity. These properties of LOCO Edit benefit greatly from the low-dimensional semantic subspace. Our method can further be extended to unsupervised or text-supervised editing in various text-to-image diffusion models (T-LOCO Edit). Finally, extensive empirical experiments demonstrate the effectiveness and efficiency of LOCO Edit.