Performance Prediction for Multi-hop Questions

TL;DR

This paper introduces multHP, a novel pre-retrieval method for predicting the difficulty of open-domain multi-hop questions, outperforming traditional models and aiding in system optimization.

Contribution

We propose multHP, a new pre-retrieval performance prediction model specifically designed for multi-hop QA, addressing the lack of existing methods for this complex task.

Findings

multHP outperforms traditional single-hop QPP models

The model effectively predicts multi-hop question difficulty

Using multHP improves retrieval system parameters and overall performance

Abstract

We study the problem of Query Performance Prediction (QPP) for open-domain multi-hop Question Answering (QA), where the task is to estimate the difficulty of evaluating a multi-hop question over a corpus. Despite the extensive research on predicting the performance of ad-hoc and QA retrieval models, there has been a lack of study on the estimation of the difficulty of multi-hop questions. The problem is challenging due to the multi-step nature of the retrieval process, potential dependency of the steps and the reasoning involved. To tackle this challenge, we propose multHP, a novel pre-retrieval method for predicting the performance of open-domain multi-hop questions. Our extensive evaluation on the largest multi-hop QA dataset using several modern QA systems shows that the proposed model is a strong predictor of the performance, outperforming traditional single-hop QPP models. Additionally, we demonstrate that our approach can be effectively used to optimize the parameters of QA systems, such as the number of documents to be retrieved, resulting in improved overall retrieval performance.