Exploring the Representation Manifolds of Stable Diffusion Through the Lens of Intrinsic Dimension

TL;DR

This paper investigates how prompts influence the geometric properties of internal representations in Stable Diffusion, revealing that prompt choice significantly affects the intrinsic dimension of model representations across different layers.

Contribution

It provides the first mathematical analysis of prompt effects on the intrinsic dimension of representations in Stable Diffusion, highlighting layer-dependent impacts.

Findings

Prompt choice significantly affects intrinsic dimension.

Intrinsic dimension correlates with prompt perplexity in bottleneck layers.

Layer-dependent variations in representation geometry were observed.

Abstract

Prompting has become an important mechanism by which users can more effectively interact with many flavors of foundation model. Indeed, the last several years have shown that well-honed prompts can sometimes unlock emergent capabilities within such models. While there has been a substantial amount of empirical exploration of prompting within the community, relatively few works have studied prompting at a mathematical level. In this work we aim to take a first step towards understanding basic geometric properties induced by prompts in Stable Diffusion, focusing on the intrinsic dimension of internal representations within the model. We find that choice of prompt has a substantial impact on the intrinsic dimension of representations at both layers of the model which we explored, but that the nature of this impact depends on the layer being considered. For example, in certain bottleneck layers of the model, intrinsic dimension of representations is correlated with prompt perplexity (measured using a surrogate model), while this correlation is not apparent in the latent layers. Our evidence suggests that intrinsic dimension could be a useful tool for future studies of the impact of different prompts on text-to-image models.