Value Function Decomposition in Markov Recommendation Process

TL;DR

This paper proposes a value function decomposition method for reinforcement learning in recommender systems, improving long-term reward estimation, learning speed, and robustness by disentangling stochastic policy and environment factors.

Contribution

It introduces a novel disentangled learning framework that separately models stochastic policy and user environment factors in value function estimation for better recommendation performance.

Findings

Faster convergence in value estimation

Enhanced robustness against action exploration

Improved long-term reward prediction accuracy

Abstract

Recent advances in recommender systems have shown that user-system interaction essentially formulates long-term optimization problems, and online reinforcement learning can be adopted to improve recommendation performance. The general solution framework incorporates a value function that estimates the user's expected cumulative rewards in the future and guides the training of the recommendation policy. To avoid local maxima, the policy may explore potential high-quality actions during inference to increase the chance of finding better future rewards. To accommodate the stepwise recommendation process, one widely adopted approach to learning the value function is learning from the difference between the values of two consecutive states of a user. However, we argue that this paradigm involves a challenge of Mixing Random Factors: there exist two random factors from the stochastic policy and the uncertain user environment, but they are not separately modeled in the standard temporal difference (TD) learning, which may result in a suboptimal estimation of the long-term rewards and less effective action exploration. As a solution, we show that these two factors can be separately approximated by decomposing the original temporal difference loss. The disentangled learning framework can achieve a more accurate estimation with faster learning and improved robustness against action exploration. As an empirical verification of our proposed method, we conduct offline experiments with simulated online environments built on the basis of public datasets.