EDITOR: Effective and Interpretable Prompt Inversion for Text-to-Image Diffusion Models

TL;DR

This paper introduces extsc{EDITR}, a novel prompt inversion method for text-to-image diffusion models that improves image similarity, interpretability, and generalizability by leveraging captioning models and latent space refinement.

Contribution

extsc{EDITR} combines caption-based initialization with latent space reverse-engineering to enhance prompt inversion for diffusion models, outperforming existing methods.

Findings

Outperforms existing prompt inversion methods in image similarity and interpretability

Effective across multiple datasets including MS COCO, LAION, Flickr, and DiffusionDB

Enables applications like cross-concept synthesis and unsupervised segmentation

Abstract

Text-to-image generation models~(e.g., Stable Diffusion) have achieved significant advancements, enabling the creation of high-quality and realistic images based on textual descriptions. Prompt inversion, the task of identifying the textual prompt used to generate a specific artifact, holds significant potential for applications including data attribution, model provenance, and watermarking validation. Recent studies introduced a delayed projection scheme to optimize for prompts representative of the vocabulary space, though challenges in semantic fluency and efficiency remain. Advanced image captioning models or visual large language models can generate highly interpretable prompts, but they often lack in image similarity. In this paper, we propose a prompt inversion technique called \sys for text-to-image diffusion models, which includes initializing embeddings using a pre-trained image captioning model, refining them through reverse-engineering in the latent space, and converting them to texts using an embedding-to-text model. Our experiments on the widely-used datasets, such as MS COCO, LAION, Flickr and DiffusionDB, show that our method outperforms existing methods in terms of image similarity, textual alignment, prompt interpretability and generalizability. We further illustrate the application of our generated prompts in tasks such as cross-concept image synthesis, concept manipulation, evolutionary multi-concept generation and unsupervised segmentation.