Breaking Flat: A Generalised Query Performance Prediction Evaluation Framework

TL;DR

This paper introduces a comprehensive framework for evaluating query performance prediction models across three different settings, highlighting the varying effectiveness and challenges in predicting query difficulty and optimal rankers.

Contribution

It generalizes the QPP evaluation to three new settings, providing insights into the varying difficulty of predicting query performance and selecting the best ranker.

Findings

Effectiveness of QPP models varies across evaluation settings.

Predicting the best ranker for a query is more challenging.

Different evaluation settings reveal different strengths of QPP models.

Abstract

The traditional use-case of query performance prediction (QPP) is to identify which queries perform well and which perform poorly for a given ranking model. A more fine-grained and arguably more challenging extension of this task is to determine which ranking models are most effective for a given query. In this work, we generalize the QPP task and its evaluation into three settings: (i) SingleRanker MultiQuery (SRMQ-PP), corresponding to the standard use case; (ii) MultiRanker SingleQuery (MRSQ-PP), which evaluates a QPP model's ability to select the most effective ranker for a query; and (iii) MultiRanker MultiQuery (MRMQ-PP), which considers predictions jointly across all query ranker pairs. Our results show that (a) the relative effectiveness of QPP models varies substantially across tasks (SRMQ-PP vs. MRSQ-PP), and (b) predicting the best ranker for a query is considerably more difficult than predicting the relative difficulty of queries for a given ranker.