VQPP: Video Query Performance Prediction Benchmark

TL;DR

This paper introduces the first benchmark for video query performance prediction (VQPP), providing datasets and evaluation tools to advance research in content-based video retrieval and enabling new applications like query reformulation.

Contribution

It presents the first VQPP benchmark with datasets, evaluation splits, and baseline predictors, facilitating reproducible research and exploration of performance prediction in video retrieval.

Findings

Pre-retrieval predictors perform competitively, enabling early prediction applications.

VQPP benchmark supports future research in video retrieval performance prediction.

Best pre-retrieval predictor can be used as a reward model for training language models.

Abstract

Query performance prediction (QPP) is an important and actively studied information retrieval task, having various applications, such as query reformulation, query expansion, and retrieval system selection, among many others. The task has been primarily studied in the context of text and image retrieval, whereas QPP for content-based video retrieval (CBVR) remains largely underexplored. To this end, we propose the first benchmark for video query performance prediction (VQPP), comprising two text-to-video retrieval datasets and two CBVR systems, respectively. VQPP contains a total of 56K text queries and 51K videos, and comes with official training, validation and test splits, fostering direct comparisons and reproducible results. We explore multiple pre-retrieval and post-retrieval performance predictors, creating a representative benchmark for future exploration of QPP in the video domain. Our results show that pre-retrieval predictors obtain competitive performance, enabling applications before performing the retrieval step. We also demonstrate the applicability of VQPP by employing the best performing pre-retrieval predictor as reward model for training a large language model (LLM) on the query reformulation task via direct preference optimization (DPO). We release our benchmark and code at https://github.com/AdrianLutu/VQPP.