RLIRank: Learning to Rank with Reinforcement Learning for Dynamic Search

TL;DR

This paper introduces RLIrank, a reinforcement learning-based dynamic ranking model that adapts search results based on user feedback and evolving information needs, outperforming previous methods in TREC benchmarks.

Contribution

The paper presents a novel reinforcement learning framework with an LSTM-based ranking model and a word-embedding Rocchio Algorithm for dynamic search tasks.

Findings

RLIrank outperforms previous methods in TREC Dynamic Domain Tracks 2016 and 2017.

The LSTM-based model effectively learns from previous rankings and user feedback.

The approach advances the state of the art in dynamic search ranking.

Abstract

To support complex search tasks, where the initial information requirements are complex or may change during the search, a search engine must adapt the information delivery as the user's information requirements evolve. To support this dynamic ranking paradigm effectively, search result ranking must incorporate both the user feedback received, and the information displayed so far. To address this problem, we introduce a novel reinforcement learning-based approach, RLIrank. We first build an adapted reinforcement learning framework to integrate the key components of the dynamic search. Then, we implement a new Learning to Rank (LTR) model for each iteration of the dynamic search, using a recurrent Long Short Term Memory neural network (LSTM), which estimates the gain for each next result, learning from each previously ranked document. To incorporate the user's feedback, we develop a word-embedding variation of the classic Rocchio Algorithm, to help guide the ranking towards the high-value documents. Those innovations enable RLIrank to outperform the previously reported methods from the TREC Dynamic Domain Tracks 2017 and exceed all the methods in 2016 TREC Dynamic Domain after multiple search iterations, advancing the state of the art for dynamic search.