Unified Off-Policy Learning to Rank: a Reinforcement Learning Perspective

TL;DR

This paper introduces CUOLR, a reinforcement learning approach that unifies off-policy learning to rank across various click models by modeling the ranking process as an MDP, enabling robust and model-agnostic learning.

Contribution

The paper proposes a novel MDP formulation for off-policy LTR that is agnostic to click models and applies offline RL techniques for improved robustness and versatility.

Findings

CUOLR outperforms existing algorithms on large-scale datasets.

It maintains robustness across different click models.

The method simplifies off-policy LTR without complex debiasing.

Abstract

Off-policy Learning to Rank (LTR) aims to optimize a ranker from data collected by a deployed logging policy. However, existing off-policy learning to rank methods often make strong assumptions about how users generate the click data, i.e., the click model, and hence need to tailor their methods specifically under different click models. In this paper, we unified the ranking process under general stochastic click models as a Markov Decision Process (MDP), and the optimal ranking could be learned with offline reinforcement learning (RL) directly. Building upon this, we leverage offline RL techniques for off-policy LTR and propose the Click Model-Agnostic Unified Off-policy Learning to Rank (CUOLR) method, which could be easily applied to a wide range of click models. Through a dedicated formulation of the MDP, we show that offline RL algorithms can adapt to various click models without complex debiasing techniques and prior knowledge of the model. Results on various large-scale datasets demonstrate that CUOLR consistently outperforms the state-of-the-art off-policy learning to rank algorithms while maintaining consistency and robustness under different click models.