Analyzing and Learning from User Interactions for Search Clarification

TL;DR

This paper analyzes large-scale user interactions with search clarifying questions to understand engagement patterns, biases, and proposes a model for re-ranking clarifying questions to improve search intent clarification.

Contribution

It provides a comprehensive analysis of user interactions with clarifying questions and introduces a novel model for re-ranking questions based on implicit feedback.

Findings

Identified position and presentation biases in user interaction data.

Demonstrated the effectiveness of the re-ranking model on click and human-labeled data.

Showed that user engagement with clarifying questions can be modeled for improved search clarification.

Abstract

Asking clarifying questions in response to search queries has been recognized as a useful technique for revealing the underlying intent of the query. Clarification has applications in retrieval systems with different interfaces, from the traditional web search interfaces to the limited bandwidth interfaces as in speech-only and small screen devices. Generation and evaluation of clarifying questions have been recently studied in the literature. However, user interaction with clarifying questions is relatively unexplored. In this paper, we conduct a comprehensive study by analyzing large-scale user interactions with clarifying questions in a major web search engine. In more detail, we analyze the user engagements received by clarifying questions based on different properties of search queries, clarifying questions, and their candidate answers. We further study click bias in the data, and show that even though reading clarifying questions and candidate answers does not take significant efforts, there still exist some position and presentation biases in the data. We also propose a model for learning representation for clarifying questions based on the user interaction data as implicit feedback. The model is used for re-ranking a number of automatically generated clarifying questions for a given query. Evaluation on both click data and human labeled data demonstrates the high quality of the proposed method.