Balancing Reinforcement Learning Training Experiences in Interactive Information Retrieval

TL;DR

This paper tackles the challenge of unbalanced training data in applying reinforcement learning to interactive information retrieval by synthesizing relevant documents through domain randomization, significantly improving learning effectiveness.

Contribution

It introduces a domain randomization approach to generate relevant training data, enhancing RL performance in IIR tasks with imbalanced relevance labels.

Findings

Boosts RL learning effectiveness by 22% on TREC DD 2017 data

Addresses sample inefficiency in RL for IIR

Improves policy learning with synthetic relevant documents

Abstract

Interactive Information Retrieval (IIR) and Reinforcement Learning (RL) share many commonalities, including an agent who learns while interacts, a long-term and complex goal, and an algorithm that explores and adapts. To successfully apply RL methods to IIR, one challenge is to obtain sufficient relevance labels to train the RL agents, which are infamously known as sample inefficient. However, in a text corpus annotated for a given query, it is not the relevant documents but the irrelevant documents that predominate. This would cause very unbalanced training experiences for the agent and prevent it from learning any policy that is effective. Our paper addresses this issue by using domain randomization to synthesize more relevant documents for the training. Our experimental results on the Text REtrieval Conference (TREC) Dynamic Domain (DD) 2017 Track show that the proposed method is able to boost an RL agent's learning effectiveness by 22\% in dealing with unseen situations.