Query Performance Prediction for Neural IR: Are We There Yet?

TL;DR

This paper evaluates how well existing Query Performance Prediction models work for neural IR systems, finding they perform significantly worse compared to traditional methods, especially in semantic-rich retrieval tasks.

Contribution

The study systematically compares QPP models on neural versus traditional IR systems, revealing their limitations in the neural IR context.

Findings

QPP models perform significantly worse on neural IR systems.

Performance drop of up to 10% in semantic retrieval tasks.

QPPs struggle to predict performance for neural IR when it differs from traditional methods.

Abstract

Evaluation in Information Retrieval relies on post-hoc empirical procedures, which are time-consuming and expensive operations. To alleviate this, Query Performance Prediction (QPP) models have been developed to estimate the performance of a system without the need for human-made relevance judgements. Such models, usually relying on lexical features from queries and corpora, have been applied to traditional sparse IR methods - with various degrees of success. With the advent of neural IR and large Pre-trained Language Models, the retrieval paradigm has significantly shifted towards more semantic signals. In this work, we study and analyze to what extent current QPP models can predict the performance of such systems. Our experiments consider seven traditional bag-of-words and seven BERT-based IR approaches, as well as nineteen state-of-the-art QPPs evaluated on two collections, Deep Learning '19 and Robust '04. Our findings show that QPPs perform statistically significantly worse on neural IR systems. In settings where semantic signals are prominent (e.g., passage retrieval), their performance on neural models drops by as much as 10% compared to bag-of-words approaches. On top of that, in lexical-oriented scenarios, QPPs fail to predict performance for neural IR systems on those queries where they differ from traditional approaches the most.