Offline Meta-level Model-based Reinforcement Learning Approach for Cold-Start Recommendation

TL;DR

This paper introduces a meta-level model-based reinforcement learning method that enables recommender systems to quickly adapt to new users with limited interactions by inferring user preferences and recovering policies efficiently.

Contribution

It proposes a novel meta-level RL approach with inverse reinforcement learning for fast cold-start user adaptation in recommender systems.

Findings

Effective adaptation to new users with only one interaction sequence

Improved recommendation performance in cold-start scenarios

Theoretical bounds on recommendation performance

Abstract

Reinforcement learning (RL) has shown great promise in optimizing long-term user interest in recommender systems. However, existing RL-based recommendation methods need a large number of interactions for each user to learn a robust recommendation policy. The challenge becomes more critical when recommending to new users who have a limited number of interactions. To that end, in this paper, we address the cold-start challenge in the RL-based recommender systems by proposing a meta-level model-based reinforcement learning approach for fast user adaptation. In our approach, we learn to infer each user's preference with a user context variable that enables recommendation systems to better adapt to new users with few interactions. To improve adaptation efficiency, we learn to recover the user policy and reward from only a few interactions via an inverse reinforcement learning method to assist a meta-level recommendation agent. Moreover, we model the interaction relationship between the user model and recommendation agent from an information-theoretic perspective. Empirical results show the effectiveness of the proposed method when adapting to new users with only a single interaction sequence. We further provide a theoretical analysis of the recommendation performance bound.