Self-Attention Decomposition For Training Free Diffusion Editing

TL;DR

This paper introduces an analytical method to derive semantic editing directions directly from pretrained diffusion models' self-attention weights, enabling efficient and high-quality image editing without additional data or training.

Contribution

The authors propose a novel approach that extracts interpretable editing directions from self-attention matrices of diffusion models, eliminating the need for sampling or auxiliary training.

Findings

Reduces editing time by 60% compared to benchmarks.

Produces high-quality, interpretable edits across multiple datasets.

Does not require additional data or model fine-tuning.

Abstract

Diffusion models achieve remarkable fidelity in image synthesis, yet precise control over their outputs for targeted editing remains challenging. A key step toward controllability is to identify interpretable directions in the model's latent representations that correspond to semantic attributes. Existing approaches for finding interpretable directions typically rely on sampling large sets of images or training auxiliary networks, which limits efficiency. We propose an analytical method that derives semantic editing directions directly from the pretrained parameters of diffusion models, requiring neither additional data nor fine-tuning. Our insight is that self-attention weight matrices encode rich structural information about the data distribution learned during training. By computing the eigenvectors of these weight matrices, we obtain robust and interpretable editing directions. Experiments demonstrate that our method produces high-quality edits across multiple datasets while reducing editing time significantly by 60% over current benchmarks.