Constrained Reinforcement Learning for Short Video Recommendation

TL;DR

This paper introduces a constrained reinforcement learning framework for short video recommendation that balances optimizing user watch time with auxiliary interactions, demonstrating superior performance in simulations and live platform deployment.

Contribution

It proposes a novel two-stage actor-critic reinforcement learning method for constrained MDPs tailored to short video recommendation, effectively balancing multiple user response objectives.

Findings

Outperforms baseline methods in simulations for main and auxiliary objectives.

Significantly improves user watch time and interactions in live experiments.

Successfully deployed in a production system for real-world short video recommendations.

Abstract

The wide popularity of short videos on social media poses new opportunities and challenges to optimize recommender systems on the video-sharing platforms. Users provide complex and multi-faceted responses towards recommendations, including watch time and various types of interactions with videos. As a result, established recommendation algorithms that concern a single objective are not adequate to meet this new demand of optimizing comprehensive user experiences. In this paper, we formulate the problem of short video recommendation as a constrained Markov Decision Process (MDP), where platforms want to optimize the main goal of user watch time in long term, with the constraint of accommodating the auxiliary responses of user interactions such as sharing/downloading videos. To solve the constrained MDP, we propose a two-stage reinforcement learning approach based on actor-critic framework. At stage one, we learn individual policies to optimize each auxiliary response. At stage two, we learn a policy to (i) optimize the main response and (ii) stay close to policies learned at the first stage, which effectively guarantees the performance of this main policy on the auxiliaries. Through extensive simulations, we demonstrate effectiveness of our approach over alternatives in both optimizing the main goal as well as balancing the others. We further show the advantage of our approach in live experiments of short video recommendations, where it significantly outperforms other baselines in terms of watch time and interactions from video views. Our approach has been fully launched in the production system to optimize user experiences on the platform.