iQPP: A Benchmark for Image Query Performance Prediction

TL;DR

This paper introduces iQPP, the first benchmark for predicting image query performance in content-based image retrieval, highlighting the challenges and gaps in current prediction methods across diverse datasets.

Contribution

The paper establishes a new benchmark with datasets, ground-truth difficulty measures, and evaluates novel and existing predictors for image query performance prediction.

Findings

Most predictors do not generalize well across scenarios

iQPP reveals significant challenges in image query performance prediction

Open source code and data are provided for future research

Abstract

To date, query performance prediction (QPP) in the context of content-based image retrieval remains a largely unexplored task, especially in the query-by-example scenario, where the query is an image. To boost the exploration of the QPP task in image retrieval, we propose the first benchmark for image query performance prediction (iQPP). First, we establish a set of four data sets (PASCAL VOC 2012, Caltech-101, ROxford5k and RParis6k) and estimate the ground-truth difficulty of each query as the average precision or the precision@k, using two state-of-the-art image retrieval models. Next, we propose and evaluate novel pre-retrieval and post-retrieval query performance predictors, comparing them with existing or adapted (from text to image) predictors. The empirical results show that most predictors do not generalize across evaluation scenarios. Our comprehensive experiments indicate that iQPP is a challenging benchmark, revealing an important research gap that needs to be addressed in future work. We release our code and data as open source at https://github.com/Eduard6421/iQPP, to foster future research.