High Quality Related Search Query Suggestions using Deep Reinforcement Learning

TL;DR

This paper introduces a deep reinforcement learning approach to generate high-quality related search query suggestions by optimizing for diversity, relevance, and user engagement, addressing limitations of previous supervised and reinforcement learning methods.

Contribution

The paper presents a novel deep reinforcement learning model that effectively incorporates long-term user feedback and syntactic relatedness to improve query suggestion quality.

Findings

3% increase in recommendation diversity

4.2% improvement in user engagement

82% reduction in word repetitions

Abstract

"High Quality Related Search Query Suggestions" task aims at recommending search queries which are real, accurate, diverse, relevant and engaging. Obtaining large amounts of query-quality human annotations is expensive. Prior work on supervised query suggestion models suffered from selection and exposure bias, and relied on sparse and noisy immediate user-feedback (e.g., clicks), leading to low quality suggestions. Reinforcement Learning techniques employed to reformulate a query using terms from search results, have limited scalability to large-scale industry applications. To recommend high quality related search queries, we train a Deep Reinforcement Learning model to predict the query a user would enter next. The reward signal is composed of long-term session-based user feedback, syntactic relatedness and estimated naturalness of generated query. Over the baseline supervised model, our proposed approach achieves a significant relative improvement in terms of recommendation diversity (3%), down-stream user-engagement (4.2%) and per-sentence word repetitions (82%).