CUPID: Contextual Understanding of Prompt-conditioned Image Distributions

TL;DR

CUPID is a visualization technique that enables detailed analysis of prompt-conditioned image distributions from generative models, revealing object styles, relationships, and biases through low-dimensional embeddings.

Contribution

The paper introduces a novel visualization method using density-based embeddings for analyzing high-dimensional image distributions conditioned on prompts.

Findings

Reveals biases in object composition in diffusion models

Identifies salient and rare object styles within generated distributions

Provides insights into language misunderstandings in generative models

Abstract

We present CUPID: a visualization method for the contextual understanding of prompt-conditioned image distributions. CUPID targets the visual analysis of distributions produced by modern text-to-image generative models, wherein a user can specify a scene via natural language, and the model generates a set of images, each intended to satisfy the user's description. CUPID is designed to help understand the resulting distribution, using contextual cues to facilitate analysis: objects mentioned in the prompt, novel, synthesized objects not explicitly mentioned, and their potential relationships. Central to CUPID is a novel method for visualizing high-dimensional distributions, wherein contextualized embeddings of objects, those found within images, are mapped to a low-dimensional space via density-based embeddings. We show how such embeddings allows one to discover salient styles of objects within a distribution, as well as identify anomalous, or rare, object styles. Moreover, we introduce conditional density embeddings, whereby conditioning on a given object allows one to compare object dependencies within the distribution. We employ CUPID for analyzing image distributions produced by large-scale diffusion models, where our experimental results offer insights on language misunderstanding from such models and biases in object composition, while also providing an interface for discovery of typical, or rare, synthesized scenes.