Beyond Correlations: A Downstream Evaluation Framework for Query Performance Prediction

TL;DR

This paper introduces a new evaluation framework for query performance prediction that focuses on downstream application effectiveness, demonstrating its practical benefits in IR fusion and highlighting limitations of traditional correlation-based metrics.

Contribution

It proposes a downstream-focused evaluation method for QPP that assesses its impact on IR fusion, providing a more application-relevant measure of predictor quality.

Findings

QPP estimates improve weighted IR fusion by over 4.5%.

Downstream effectiveness of QPP does not correlate with standard correlation metrics.

Using QPP as priors enhances IR fusion performance.

Abstract

The standard practice of query performance prediction (QPP) evaluation is to measure a set-level correlation between the estimated retrieval qualities and the true ones. However, neither this correlation-based evaluation measure quantifies QPP effectiveness at the level of individual queries, nor does this connect to a downstream application, meaning that QPP methods yielding high correlation values may not find a practical application in query-specific decisions in an IR pipeline. In this paper, we propose a downstream-focussed evaluation framework where a distribution of QPP estimates across a list of top-documents retrieved with several rankers is used as priors for IR fusion. While on the one hand, a distribution of these estimates closely matching that of the true retrieval qualities indicates the quality of the predictor, their usage as priors on the other hand indicates a predictor's ability to make informed choices in an IR pipeline. Our experiments firstly establish the importance of QPP estimates in weighted IR fusion, yielding substantial improvements of over 4.5% over unweighted CombSUM and RRF fusion strategies, and secondly, reveal new insights that the downstream effectiveness of QPP does not correlate well with the standard correlation-based QPP evaluation.