Model-free Reinforcement Learning with Stochastic Reward Stabilization for Recommender Systems

TL;DR

This paper addresses the challenge of stochastic user feedback in model-free reinforcement learning for recommender systems by proposing two stabilization frameworks that improve performance.

Contribution

It introduces two novel, model-agnostic frameworks for stabilizing stochastic rewards in RL-based recommender systems, enhancing their effectiveness.

Findings

Proposed frameworks outperform baseline methods in simulations.

Frameworks effectively utilize various supervised models.

Significant performance improvements in industrial recommender systems.

Abstract

Model-free RL-based recommender systems have recently received increasing research attention due to their capability to handle partial feedback and long-term rewards. However, most existing research has ignored a critical feature in recommender systems: one user's feedback on the same item at different times is random. The stochastic rewards property essentially differs from that in classic RL scenarios with deterministic rewards, which makes RL-based recommender systems much more challenging. In this paper, we first demonstrate in a simulator environment where using direct stochastic feedback results in a significant drop in performance. Then to handle the stochastic feedback more efficiently, we design two stochastic reward stabilization frameworks that replace the direct stochastic feedback with that learned by a supervised model. Both frameworks are model-agnostic, i.e., they can effectively utilize various supervised models. We demonstrate the superiority of the proposed frameworks over different RL-based recommendation baselines with extensive experiments on a recommendation simulator as well as an industrial-level recommender system.