An IR-based Evaluation Framework for Web Search Query Segmentation

TL;DR

This paper introduces an IR-based evaluation framework for web search query segmentation, revealing that algorithm effectiveness often surpasses human annotations in IR performance, and provides insights for improving segmentation strategies.

Contribution

It presents the first IR-based evaluation framework for query segmentation, challenging traditional annotation-based validation and offering a new perspective on segmentation effectiveness.

Findings

State-of-the-art algorithms match or outperform human annotations in IR tasks.

The evaluation framework identifies key segments necessary for optimal retrieval.

The dataset used for evaluation is publicly available for research use.

Abstract

This paper presents the first evaluation framework for Web search query segmentation based directly on IR performance. In the past, segmentation strategies were mainly validated against manual annotations. Our work shows that the goodness of a segmentation algorithm as judged through evaluation against a handful of human annotated segmentations hardly reflects its effectiveness in an IR-based setup. In fact, state-of the-art algorithms are shown to perform as good as, and sometimes even better than human annotations -- a fact masked by previous validations. The proposed framework also provides us an objective understanding of the gap between the present best and the best possible segmentation algorithm. We draw these conclusions based on an extensive evaluation of six segmentation strategies, including three most recent algorithms, vis-a-vis segmentations from three human annotators. The evaluation framework also gives insights about which segments should be necessarily detected by an algorithm for achieving the best retrieval results. The meticulously constructed dataset used in our experiments has been made public for use by the research community.