CTR is not Enough: a Novel Reinforcement Learning based Ranking Approach for Optimizing Session Clicks

TL;DR

This paper introduces a reinforcement learning-based ranking method that optimizes long-term user engagement by considering user leaving behavior, outperforming traditional CTR-focused approaches in recommendation systems.

Contribution

It proposes a novel long-term optimization goal called CTE and develops a reinforcement learning model to improve session-level recommendation performance.

Findings

Significant improvement in session click counts in offline tests

Enhanced online performance in TaoBao e-commerce platform

Effective modeling of user leaving behavior

Abstract

Ranking is a crucial module using in the recommender system. In particular, the ranking module using in our YoungTao recommendation scenario is to provide an ordered list of items to users, to maximize the click number throughout the recommendation session for each user. However, we found that the traditional ranking method for optimizing Click-Through rate(CTR) cannot address our ranking scenario well, since it completely ignores user leaving, and CTR is the optimization goal for the one-step recommendation. To effectively undertake the purpose of our ranking module, we propose a long-term optimization goal, named as CTE (Click-Through quantity expectation), for explicitly taking the behavior of user leaving into account. Based on CTE, we propose an effective model trained by reinforcement learning. Moreover, we build a simulation environment from offline log data for estimating PBR and CTR. We conduct extensive experiments on offline datasets and an online e-commerce scenario in TaoBao. Experimental results show that our method can boost performance effectively