PQPP: A Joint Benchmark for Text-to-Image Prompt and Query Performance Prediction

TL;DR

This paper introduces a comprehensive benchmark dataset for predicting the difficulty of prompts in text-to-image generation and retrieval, enabling better assessment and development of performance predictors for both tasks.

Contribution

It creates the first joint benchmark for prompt and query performance prediction in text-to-image generation and retrieval, with manually annotated data for over 10,000 queries.

Findings

Benchmark enables comparison of prompt/query difficulty in both tasks.

Evaluates several performance predictors, establishing baselines.

Provides publicly available dataset and code for future research.

Abstract

Text-to-image generation has recently emerged as a viable alternative to text-to-image retrieval, driven by the visually impressive results of generative diffusion models. Although query performance prediction is an active research topic in information retrieval, to the best of our knowledge, there is no prior study that analyzes the difficulty of queries (referred to as prompts) in text-to-image generation, based on human judgments. To this end, we introduce the first dataset of prompts which are manually annotated in terms of image generation performance. Additionally, we extend these evaluations to text-to-image retrieval by collecting manual annotations that represent retrieval performance. We thus establish the first joint benchmark for prompt and query performance prediction (PQPP) across both tasks, comprising over 10K queries. Our benchmark enables (i) the comparative assessment of prompt/query difficulty in both image generation and image retrieval, and (ii) the evaluation of prompt/query performance predictors addressing both generation and retrieval. We evaluate several pre- and post-generation/retrieval performance predictors, thus providing competitive baselines for future research. Our benchmark and code are publicly available at https://github.com/Eduard6421/PQPP.