Circuit Mechanisms for Spatial Relation Generation in Diffusion Transformers

TL;DR

This paper investigates how diffusion transformers generate spatial relations in images, revealing different underlying mechanisms depending on the text encoder used, with implications for robustness.

Contribution

It uncovers distinct circuit mechanisms in diffusion transformers for spatial relation generation based on text encoder choice, advancing mechanistic interpretability.

Findings

Two-stage circuit with separate attention heads when using random embeddings.

Single-text-token fusion circuit when using pretrained T5 encoder.

Different robustness to out-of-domain perturbations based on encoder choice.

Abstract

Diffusion Transformers (DiTs) have greatly advanced text-to-image generation, but models still struggle to generate the correct spatial relations between objects as specified in the text prompt. In this study, we adopt a mechanistic interpretability approach to investigate how a DiT can generate correct spatial relations between objects. We train, from scratch, DiTs of different sizes with different text encoders to learn to generate images containing two objects whose attributes and spatial relations are specified in the text prompt. We find that, although all the models can learn this task to near-perfect accuracy, the underlying mechanisms differ drastically depending on the choice of text encoder. When using random text embeddings, we find that the spatial-relation information is passed to image tokens through a two-stage circuit, involving two cross-attention heads that separately read the spatial relation and single-object attributes in the text prompt. When using a pretrained text encoder (T5), we find that the DiT uses a different circuit that leverages information fusion in the text tokens, reading spatial-relation and single-object information together from a single text token. We further show that, although the in-domain performance is similar for the two settings, their robustness to out-of-domain perturbations differs, potentially suggesting the difficulty of generating correct relations in real-world scenarios.