Supervised Advantage Actor-Critic for Recommender Systems

TL;DR

This paper introduces a novel RL-based framework for sequential recommendation that combines supervised learning with advantage actor-critic methods, addressing key challenges like biased Q-value estimation and large action spaces.

Contribution

It proposes the Supervised Advantage Actor-Critic (SA2C) framework with a negative sampling strategy, improving recommendation performance over existing methods.

Findings

Significantly outperforms state-of-the-art supervised methods.

Achieves better results than existing self-supervised RL approaches.

Demonstrates effectiveness on real-world datasets.

Abstract

Casting session-based or sequential recommendation as reinforcement learning (RL) through reward signals is a promising research direction towards recommender systems (RS) that maximize cumulative profits. However, the direct use of RL algorithms in the RS setting is impractical due to challenges like off-policy training, huge action spaces and lack of sufficient reward signals. Recent RL approaches for RS attempt to tackle these challenges by combining RL and (self-)supervised sequential learning, but still suffer from certain limitations. For example, the estimation of Q-values tends to be biased toward positive values due to the lack of negative reward signals. Moreover, the Q-values also depend heavily on the specific timestamp of a sequence. To address the above problems, we propose negative sampling strategy for training the RL component and combine it with supervised sequential learning. We call this method Supervised Negative Q-learning (SNQN). Based on sampled (negative) actions (items), we can calculate the "advantage" of a positive action over the average case, which can be further utilized as a normalized weight for learning the supervised sequential part. This leads to another learning framework: Supervised Advantage Actor-Critic (SA2C). We instantiate SNQN and SA2C with four state-of-the-art sequential recommendation models and conduct experiments on two real-world datasets. Experimental results show that the proposed approaches achieve significantly better performance than state-of-the-art supervised methods and existing self-supervised RL methods . Code will be open-sourced.