ELROND: Exploring and decomposing intrinsic capabilities of diffusion models

TL;DR

This paper introduces a framework to analyze and control the semantic variations in diffusion model outputs by disentangling underlying generative directions, improving interpretability and diversity.

Contribution

It proposes a method to identify and manipulate semantic directions in diffusion models' input space, enhancing control and diversity of generated images.

Findings

Isolates interpretable, steerable directions for concept control

Reduces mode collapse and restores diversity in models

Provides a new estimator for concept complexity based on subspace dimensionality

Abstract

A single text prompt passed to a diffusion model often yields a wide range of visual outputs determined solely by stochastic process, leaving users with no direct control over which specific semantic variations appear in the image. While existing unsupervised methods attempt to analyze these variations via output features, they omit the underlying generative process. In this work, we propose a framework to disentangle these semantic directions directly within the input embedding space. To that end, we collect a set of gradients obtained by backpropagating the differences between stochastic realizations of a fixed prompt that we later decompose into meaningful steering directions with either Principal Components Analysis or Sparse Autoencoder. Our approach yields three key contributions: (1) it isolates interpretable, steerable directions for precise, fine-grained control over a single concept; (2) it effectively mitigates mode collapse in distilled models by reintroducing lost diversity; and (3) it establishes a novel estimator for concept complexity under a specific model, based on the dimensionality of the discovered subspace.