On Coherence-based Predictors for Dense Query Performance Prediction

TL;DR

This paper introduces coherence-based predictors using neural embeddings to improve query performance prediction for dense retrieval models, revealing the influence of query types on predictor stability.

Contribution

It develops novel unsupervised coherence-based predictors tailored for dense retrieval models and analyzes how query types affect predictor performance and stability.

Findings

Achieved up to 92% accuracy improvement for TCT-ColBERT and 188% for ANCE over sparse variants.

Identified query types significantly influence query performance and predictor stability.

Dense retrieval models exhibit unique sensitivity to query type variations.

Abstract

Query Performance Prediction (QPP) estimates the effectiveness of a search engine's results in response to a query without relevance judgments. Traditionally, post-retrieval predictors have focused upon either the distribution of the retrieval scores, or the coherence of the top-ranked documents using traditional bag-of-words index representations. More recently, BERT-based models using dense embedded document representations have been used to create new predictors, but mostly applied to predict the performance of rankings created by BM25. Instead, we aim to predict the effectiveness of rankings created by single-representation dense retrieval models (ANCE & TCT-ColBERT). Therefore, we propose a number of variants of existing unsupervised coherence-based predictors that employ neural embedding representations. In our experiments on the TREC Deep Learning Track datasets, we demonstrate improved accuracy upon dense retrieval (up to 92% compared to sparse variants for TCT-ColBERT and 188% for ANCE). Going deeper, we select the most representative and best performing predictors to study the importance of differences among predictors and query types on query performance. Using existing distribution-based evaluation QPP measures and a particular type of linear mixed models, we find that query types further significantly influence query performance (and are up to 35% responsible for the unstable performance of QPP predictors), and that this sensitivity is unique to dense retrieval models. Our approach introduces a new setting for obtaining richer information on query differences in dense QPP that can explain potential unstable performance of existing predictors and outlines the unique characteristics of different query types on dense retrieval models.