Hierarchical Reinforcement Learning for Modeling User Novelty-Seeking Intent in Recommender Systems

TL;DR

This paper introduces a hierarchical reinforcement learning approach to model and adapt to users' layered novelty-seeking behaviors in recommender systems, improving long-term user engagement by balancing exploration and preference.

Contribution

The paper presents a novel hierarchical RL method that explicitly models static and dynamic aspects of user novelty-seeking intent for personalized recommendations.

Findings

Hierarchical RL effectively captures user novelty-seeking intent.

The proposed method outperforms state-of-the-art baselines.

Incorporating diversity and novelty in reward improves exploration.

Abstract

Recommending novel content, which expands user horizons by introducing them to new interests, has been shown to improve users' long-term experience on recommendation platforms \cite{chen2021values}. Users however are not constantly looking to explore novel content. It is therefore crucial to understand their novelty-seeking intent and adjust the recommendation policy accordingly. Most existing literature models a user's propensity to choose novel content or to prefer a more diverse set of recommendations at individual interactions. Hierarchical structure, on the other hand, exists in a user's novelty-seeking intent, which is manifested as a static and intrinsic user preference for seeking novelty along with a dynamic session-based propensity. To this end, we propose a novel hierarchical reinforcement learning-based method to model the hierarchical user novelty-seeking intent, and to adapt the recommendation policy accordingly based on the extracted user novelty-seeking propensity. We further incorporate diversity and novelty-related measurement in the reward function of the hierarchical RL (HRL) agent to encourage user exploration \cite{chen2021values}. We demonstrate the benefits of explicitly modeling hierarchical user novelty-seeking intent in recommendations through extensive experiments on simulated and real-world datasets. In particular, we demonstrate that the effectiveness of our proposed hierarchical RL-based method lies in its ability to capture such hierarchically-structured intent. As a result, the proposed HRL model achieves superior performance on several public datasets, compared with state-of-art baselines.