Interpreting the Weight Space of Customized Diffusion Models

TL;DR

This paper explores the weight space of fine-tuned diffusion models, demonstrating its potential as an interpretable space for generating, editing, and inverting visual identities and concepts.

Contribution

It introduces a large dataset of customized diffusion models and models the weight space as a manifold, enabling new applications like identity synthesis and semantic editing.

Findings

Sampling from the space creates new identities.

Linear directions enable semantic edits of identities.

Inversion encodes identities from out-of-distribution images.

Abstract

We investigate the space of weights spanned by a large collection of customized diffusion models. We populate this space by creating a dataset of over 60,000 models, each of which is a base model fine-tuned to insert a different person's visual identity. We model the underlying manifold of these weights as a subspace, which we term weights2weights. We demonstrate three immediate applications of this space that result in new diffusion models -- sampling, editing, and inversion. First, sampling a set of weights from this space results in a new model encoding a novel identity. Next, we find linear directions in this space corresponding to semantic edits of the identity (e.g., adding a beard), resulting in a new model with the original identity edited. Finally, we show that inverting a single image into this space encodes a realistic identity into a model, even if the input image is out of distribution (e.g., a painting). We further find that these linear properties of the diffusion model weight space extend to other visual concepts. Our results indicate that the weight space of fine-tuned diffusion models can behave as an interpretable meta-latent space producing new models.