Reward-Agnostic Prompt Optimization for Text-to-Image Diffusion Models

TL;DR

RATTPO is a flexible, reward-agnostic prompt optimization method for text-to-image diffusion models that improves prompts across various reward scenarios without needing reward-specific adjustments, enhancing efficiency and performance.

Contribution

Introduces RATTPO, a novel test-time prompt optimization approach that is adaptable to multiple reward models without modification, outperforming existing methods in efficiency and effectiveness.

Findings

RATTPO improves prompt quality across diverse reward models.

It runs 4.8 times faster than naive search baselines.

Achieves comparable performance to reward-specific fine-tuning with sufficient inference budget.

Abstract

We investigate a general approach for improving user prompts in text-to-image (T2I) diffusion models by finding prompts that maximize a reward function specified at test-time. Although diverse reward models are used for evaluating image generation, existing automated prompt engineering methods typically target specific reward configurations. Consequently, these specialized designs exhibit suboptimal performance when applied to new prompt engineering scenarios involving different reward models. To address this limitation, we introduce RATTPO (Reward-Agnostic Test-Time Prompt Optimization), a flexible test-time optimization method applicable across various reward scenarios without modification. RATTPO iteratively searches for optimized prompts by querying large language models (LLMs) \textit{without} requiring reward-specific task descriptions. Instead, it uses the optimization trajectory and a novel reward-aware feedback signal (termed a "hint") as context. Empirical results demonstrate the versatility of RATTPO, effectively enhancing user prompts across diverse reward setups that assess various generation aspects, such as aesthetics, general human preference, or spatial relationships between objects. RATTPO surpasses other test-time search baselines in search efficiency, running 4.8 times faster than naive reward-agnostic test-time search baseline on average. Furthermore, with sufficient inference budget, it can achieve comparable performance to learning-based baselines that require reward-specific fine-tuning. The code is available at https://github.com/seminkim/RATTPO.