Query Segmentation for Relevance Ranking in Web Search

TL;DR

This paper introduces a novel query segmentation approach using re-ranking with generative and discriminative models, significantly enhancing relevance ranking in web search across multiple models and datasets.

Contribution

It presents a new re-ranking based query segmentation method and a combined query representation approach that improves relevance ranking performance.

Findings

Significant improvement in relevance ranking for BM25, n-gram, and dependency models.

Effective query segmentation enhances search relevance across large datasets.

Re-ranking approach successfully applied to query segmentation, a novel application.

Abstract

In this paper, we try to answer the question of how to improve the state-of-the-art methods for relevance ranking in web search by query segmentation. Here, by query segmentation it is meant to segment the input query into segments, typically natural language phrases, so that the performance of relevance ranking in search is increased. We propose employing the re-ranking approach in query segmentation, which first employs a generative model to create top $k$ candidates and then employs a discriminative model to re-rank the candidates to obtain the final segmentation result. The method has been widely utilized for structure prediction in natural language processing, but has not been applied to query segmentation, as far as we know. Furthermore, we propose a new method for using the result of query segmentation in relevance ranking, which takes both the original query words and the segmented query phrases as units of query representation. We investigate whether our method can improve three relevance models, namely BM25, key n-gram model, and dependency model. Our experimental results on three large scale web search datasets show that our method can indeed significantly improve relevance ranking in all the three cases.