MBCAL: Sample Efficient and Variance Reduced Reinforcement Learning for Recommender Systems

TL;DR

This paper introduces MBCAL, a model-based reinforcement learning approach for recommender systems that improves sample efficiency and reduces variance by using environment modeling and counterfactual comparisons, enabling safer and more effective long-term utility optimization.

Contribution

The paper proposes MBCAL, a novel model-based RL method tailored for recommender systems, incorporating environment and advantage models with counterfactuals to enhance efficiency and stability.

Findings

MBCAL achieves higher sample efficiency than existing methods.

It significantly reduces variance in learning the future advantage.

Experimental results outperform supervised and RL-based baselines.

Abstract

In recommender systems such as news feed stream, it is essential to optimize the long-term utilities in the continuous user-system interaction processes. Previous works have proved the capability of reinforcement learning in this problem. However, there are many practical challenges to implement deep reinforcement learning in online systems, including low sample efficiency, uncontrollable risks, and excessive variances. To address these issues, we propose a novel reinforcement learning method, namely model-based counterfactual advantage learning (MBCAL). The proposed method takes advantage of the characteristics of recommender systems and draws ideas from the model-based reinforcement learning method for higher sample efficiency. It has two components: an environment model that predicts the instant user behavior one-by-one in an auto-regressive form, and a future advantage model that predicts the future utility. To alleviate the impact of excessive variance when learning the future advantage model, we employ counterfactual comparisons derived from the environment model. In consequence, the proposed method possesses high sample efficiency and significantly lower variance; Also, it is able to use existing user logs to avoid the risks of starting from scratch. In contrast to its capability, its implementation cost is relatively low, which fits well with practical systems. Theoretical analysis and elaborate experiments are presented. Results show that the proposed method transcends the other supervised learning and RL-based methods in both sample efficiency and asymptotic performances.