Unsupervised Discovery of Semantic Latent Directions in Diffusion Models

TL;DR

This paper introduces an unsupervised approach to identify interpretable semantic directions in the latent space of diffusion models, enhancing understanding and editing capabilities without supervision.

Contribution

It proposes a novel Riemannian geometry-based method to discover semantic directions in diffusion models' latent space, revealing disentangled attributes and their geometric properties.

Findings

Semantic directions yield disentangled attribute changes

Editing at different timesteps affects different attribute levels

Method is effective across various datasets and models

Abstract

Despite the success of diffusion models (DMs), we still lack a thorough understanding of their latent space. While image editing with GANs builds upon latent space, DMs rely on editing the conditions such as text prompts. We present an unsupervised method to discover interpretable editing directions for the latent variables $\mathbf{x}_t \in \mathcal{X}$ of DMs. Our method adopts Riemannian geometry between $\mathcal{X}$ and the intermediate feature maps $\mathcal{H}$ of the U-Nets to provide a deep understanding over the geometrical structure of $\mathcal{X}$. The discovered semantic latent directions mostly yield disentangled attribute changes, and they are globally consistent across different samples. Furthermore, editing in earlier timesteps edits coarse attributes, while ones in later timesteps focus on high-frequency details. We define the curvedness of a line segment between samples to show that $\mathcal{X}$ is a curved manifold. Experiments on different baselines and datasets demonstrate the effectiveness of our method even on Stable Diffusion. Our source code will be publicly available for the future researchers.