Hierarchical Reinforcement Learning for Temporal Abstraction of Listwise Recommendation

TL;DR

This paper introduces mccHRL, a hierarchical reinforcement learning framework for listwise recommendation systems that effectively captures user perception evolution and interest shifts, improving performance over existing methods.

Contribution

The paper proposes a novel hierarchical reinforcement learning framework with distinct temporal abstractions for listwise recommendation, addressing long-term perception and short-term interests.

Findings

Significant performance improvements over baseline methods.

Effective modeling of user perception evolution.

Validated through simulator and real-world datasets.

Abstract

Modern listwise recommendation systems need to consider both long-term user perceptions and short-term interest shifts. Reinforcement learning can be applied on recommendation to study such a problem but is also subject to large search space, sparse user feedback and long interactive latency. Motivated by recent progress in hierarchical reinforcement learning, we propose a novel framework called mccHRL to provide different levels of temporal abstraction on listwise recommendation. Within the hierarchical framework, the high-level agent studies the evolution of user perception, while the low-level agent produces the item selection policy by modeling the process as a sequential decision-making problem. We argue that such framework has a well-defined decomposition of the outra-session context and the intra-session context, which are encoded by the high-level and low-level agents, respectively. To verify this argument, we implement both a simulator-based environment and an industrial dataset-based experiment. Results observe significant performance improvement by our method, compared with several well-known baselines. Data and codes have been made public.