Analyzing Coherency in Facet-based Clarification Prompt Generation for Search

TL;DR

This paper investigates the importance of facet coherency in search clarification prompts, highlighting the limitations of current evaluation methods and proposing a classifier to assess facet quality.

Contribution

It introduces a coherency classifier for facets and demonstrates the significance of facet quality in search clarification, addressing a gap in existing evaluation procedures.

Findings

Existing metrics poorly correlate with facet coherency.

Incoherent facets are prevalent in standard datasets.

A new classifier effectively assesses facet coherency.

Abstract

Clarifying user's information needs is an essential component of modern search systems. While most of the approaches for constructing clarifying prompts rely on query facets, the impact of the quality of the facets is relatively unexplored. In this work, we concentrate on facet quality through the notion of facet coherency and assess its importance for overall usefulness for clarification in search. We find that existing evaluation procedures do not account for facet coherency, as evident by the poor correlation of coherency with automated metrics. Moreover, we propose a coherency classifier and assess the prevalence of incoherent facets in a well-established dataset on clarification. Our findings can serve as motivation for future work on the topic.