PromptEcho: Annotation-Free Reward from Vision-Language Models for Text-to-Image Reinforcement Learning

TL;DR

PromptEcho offers an annotation-free, efficient reward method for text-to-image reinforcement learning by leveraging frozen vision-language models to assess image-text alignment, improving prompt-following without additional training.

Contribution

It introduces PromptEcho, a novel reward construction approach that requires no annotation or reward model training, utilizing pre-trained VLMs for improved RL performance.

Findings

PromptEcho significantly outperforms inference-based scoring methods.

Reward quality improves with larger VLMs.

Achieves substantial gains on DenseAlignBench and other benchmarks.

Abstract

Reinforcement learning (RL) can improve the prompt following capability of text-to-image (T2I) models, yet obtaining high-quality reward signals remains challenging: CLIP Score is too coarse-grained, while VLM-based reward models (e.g., RewardDance) require costly human-annotated preference data and additional fine-tuning. We propose PromptEcho, a reward construction method that requires \emph{no} annotation and \emph{no} reward model training. Given a generated image and a guiding query, PromptEcho computes the token-level cross-entropy loss of a frozen VLM with the original prompt as the label, directly extracting the image-text alignment knowledge encoded during VLM pretraining. The reward is deterministic, computationally efficient, and improves automatically as stronger open-source VLMs become available. For evaluation, we develop DenseAlignBench, a benchmark of concept-rich dense captions for rigorously testing prompt following capability. Experimental results on two state-of-the-art T2I models (Z-Image and QwenImage-2512) demonstrate that PromptEcho achieves substantial improvements on DenseAlignBench (+26.8pp / +16.2pp net win rate), along with consistent gains on GenEval, DPG-Bench, and TIIFBench without any task-specific training. Ablation studies confirm that PromptEcho comprehensively outperforms inference-based scoring with the same VLM, and that reward quality scales with VLM size. We will open-source the trained models and the DenseAlignBench.