PromptPaint: Steering Text-to-Image Generation Through Paint Medium-like Interactions

TL;DR

PromptPaint introduces an interactive method for guiding text-to-image generation by allowing users to iteratively apply prompts to different areas and stages, similar to painting, enhancing control over challenging concepts.

Contribution

This work presents PromptPaint, a novel interactive system that enables layered, paint-like prompt editing for more intuitive and precise control in diffusion-based image generation.

Findings

Users can effectively mix prompts for complex concepts.

PromptPaint supports iterative, region-specific prompt editing.

Design trade-offs and socio-technical challenges are identified.

Abstract

While diffusion-based text-to-image (T2I) models provide a simple and powerful way to generate images, guiding this generation remains a challenge. For concepts that are difficult to describe through language, users may struggle to create prompts. Moreover, many of these models are built as end-to-end systems, lacking support for iterative shaping of the image. In response, we introduce PromptPaint, which combines T2I generation with interactions that model how we use colored paints. PromptPaint allows users to go beyond language to mix prompts that express challenging concepts. Just as we iteratively tune colors through layered placements of paint on a physical canvas, PromptPaint similarly allows users to apply different prompts to different canvas areas and times of the generative process. Through a set of studies, we characterize different approaches for mixing prompts, design trade-offs, and socio-technical challenges for generative models. With PromptPaint we provide insight into future steerable generative tools.