Ranking Clarifying Questions Based on Predicted User Engagement

TL;DR

This paper proposes a method to predict user engagement with clarification questions using lexical features, and demonstrates that these predictions can effectively improve the ranking of clarification panes in search systems.

Contribution

It introduces a neural approach to predict user engagement from lexical data and shows this improves clarification pane ranking over naive baselines.

Findings

Predicted user engagement significantly outperforms naive baselines.

Incorporating engagement predictions improves ranking metrics like NDCG and MRR.

The approach serves as a baseline for future research in clarification question ranking.

Abstract

To improve online search results, clarification questions can be used to elucidate the information need of the user. This research aims to predict the user engagement with the clarification pane as an indicator of relevance based on the lexical information: query, question, and answers. Subsequently, the predicted user engagement can be used as a feature to rank the clarification panes. Regression and classification are applied for predicting user engagement and compared to naive heuristic baselines (e.g. mean) on the new MIMICS dataset [20]. An ablation study is carried out using a RankNet model to determine whether the predicted user engagement improves clarification pane ranking performance. The prediction models were able to improve significantly upon the naive baselines, and the predicted user engagement feature significantly improved the RankNet results in terms of NDCG and MRR. This research demonstrates the potential for ranking clarification panes based on lexical information only and can serve as a first neural baseline for future research to improve on. The code is available online.