Unsupervised Discovery of Interpretable Directions in h-space of Pre-trained Diffusion Models

TL;DR

This paper introduces an unsupervised, learning-based approach to discover interpretable directions in the latent space of pre-trained diffusion models, enabling meaningful manipulations without supervision.

Contribution

It presents a novel VRAM-efficient training algorithm and a method to identify disentangled, interpretable directions in diffusion models' h-space without additional procedures.

Findings

Successfully discovers global, scalable directions in diffusion models

Maintains sample fidelity during manipulations

Effective across various datasets

Abstract

We propose the first unsupervised and learning-based method to identify interpretable directions in h-space of pre-trained diffusion models. Our method is derived from an existing technique that operates on the GAN latent space. Specifically, we employ a shift control module that works on h-space of pre-trained diffusion models to manipulate a sample into a shifted version of itself, followed by a reconstructor to reproduce both the type and the strength of the manipulation. By jointly optimizing them, the model will spontaneously discover disentangled and interpretable directions. To prevent the discovery of meaningless and destructive directions, we employ a discriminator to maintain the fidelity of shifted sample. Due to the iterative generative process of diffusion models, our training requires a substantial amount of GPU VRAM to store numerous intermediate tensors for back-propagating gradient. To address this issue, we propose a general VRAM-efficient training algorithm based on gradient checkpointing technique to back-propagate any gradient through the whole generative process, with acceptable occupancy of VRAM and sacrifice of training efficiency. Compared with existing related works on diffusion models, our method inherently identifies global and scalable directions, without necessitating any other complicated procedures. Extensive experiments on various datasets demonstrate the effectiveness of our method.