Reward Shaping for User Satisfaction in a REINFORCE Recommender

TL;DR

This paper proposes a reward shaping method in reinforcement learning recommenders that uses satisfaction measurement and imputation to improve user satisfaction, validated through offline and live experiments.

Contribution

It introduces a joint learning framework combining a satisfaction imputation network with a REINFORCE-based policy for better user satisfaction in recommendations.

Findings

Imputation models effectively predict satisfaction for unobserved interactions.

Reward shaping with satisfaction signals improves recommendation satisfaction.

Live experiments show increased user satisfaction in industrial settings.

Abstract

How might we design Reinforcement Learning (RL)-based recommenders that encourage aligning user trajectories with the underlying user satisfaction? Three research questions are key: (1) measuring user satisfaction, (2) combatting sparsity of satisfaction signals, and (3) adapting the training of the recommender agent to maximize satisfaction. For measurement, it has been found that surveys explicitly asking users to rate their experience with consumed items can provide valuable orthogonal information to the engagement/interaction data, acting as a proxy to the underlying user satisfaction. For sparsity, i.e, only being able to observe how satisfied users are with a tiny fraction of user-item interactions, imputation models can be useful in predicting satisfaction level for all items users have consumed. For learning satisfying recommender policies, we postulate that reward shaping in RL recommender agents is powerful for driving satisfying user experiences. Putting everything together, we propose to jointly learn a policy network and a satisfaction imputation network: The role of the imputation network is to learn which actions are satisfying to the user; while the policy network, built on top of REINFORCE, decides which items to recommend, with the reward utilizing the imputed satisfaction. We use both offline analysis and live experiments in an industrial large-scale recommendation platform to demonstrate the promise of our approach for satisfying user experiences.