Towards Semantic Query Segmentation

TL;DR

This paper introduces a supervised, embedding-based method for query segmentation that is fast, domain-agnostic, and achieves accuracy comparable to state-of-the-art techniques without relying on external knowledge bases.

Contribution

It presents a novel low-dimensional embedding approach for query segmentation that eliminates the need for hand-tuned features and external knowledge, enabling cross-domain applicability.

Findings

Achieves comparable accuracy to state-of-the-art on web search queries

Demonstrates effectiveness on eCommerce queries from eBay logs

Offers a fast, easy-to-implement method that generalizes across domains

Abstract

Query Segmentation is one of the critical components for understanding users' search intent in Information Retrieval tasks. It involves grouping tokens in the search query into meaningful phrases which help downstream tasks like search relevance and query understanding. In this paper, we propose a novel approach to segment user queries using distributed query embeddings. Our key contribution is a supervised approach to the segmentation task using low-dimensional feature vectors for queries, getting rid of traditional hand tuned and heuristic NLP features which are quite expensive. We benchmark on a 50,000 human-annotated web search engine query corpus achieving comparable accuracy to state-of-the-art techniques. The advantage of our technique is its fast and does not use external knowledge-base like Wikipedia for score boosting. This helps us generalize our approach to other domains like eCommerce without any fine-tuning. We demonstrate the effectiveness of this method on another 50,000 human-annotated eCommerce query corpus from eBay search logs. Our approach is easy to implement and generalizes well across different search domains proving the power of low-dimensional embeddings in query segmentation task, opening up a new direction of research for this problem.