On the Feasibility and Robustness of Pointwise Evaluation of Query Performance Prediction

TL;DR

This paper introduces a pointwise evaluation framework for query performance prediction that assesses individual query quality, offering more precise and less variable evaluations compared to traditional listwise methods.

Contribution

The paper proposes a novel pointwise evaluation method for QPP systems that enables individual query assessment and reduces variance in evaluation results.

Findings

Smaller variance in QPP evaluation results.

Effective for multiple target metrics and retrieval models.

Improves assessment precision for individual queries.

Abstract

Despite the retrieval effectiveness of queries being mutually independent of one another, the evaluation of query performance prediction (QPP) systems has been carried out by measuring rank correlation over an entire set of queries. Such a listwise approach has a number of disadvantages, notably that it does not support the common requirement of assessing QPP for individual queries. In this paper, we propose a pointwise QPP framework that allows us to evaluate the quality of a QPP system for individual queries by measuring the deviations between each prediction versus the corresponding true value, and then aggregating the results over a set of queries. Our experiments demonstrate that this new approach leads to smaller variances in QPP evaluations across a range of different target metrics and retrieval models.