Reinforce Lifelong Interaction Value of User-Author Pairs for Large-Scale Recommendation Systems

TL;DR

This paper introduces RLIV-UA, a reinforcement learning framework that optimizes lifelong user-author interaction value in large-scale recommendation systems, enhancing long-term engagement and platform profitability.

Contribution

The paper proposes a novel RL-based approach with SCRI-MDP, ASA, and MTCL to effectively model and optimize long-term user-author interactions in large-scale RS.

Findings

Higher user satisfaction in offline experiments

Increased platform profits in online A/B tests

Effective modeling of sparse interaction signals

Abstract

Recommendation systems (RS) help users find interested content and connect authors with their target audience. Most research in RS tends to focus either on predicting users' immediate feedback (like click-through rate) accurately or improving users' long-term engagement. However, they ignore the influence for authors and the lifelong interaction value (LIV) of user-author pairs, which is particularly crucial for improving the prosperity of social community in short-video platforms. Currently, reinforcement learning (RL) can optimize long-term benefits and has been widely applied in RS. In this paper, we introduce RL to Reinforce Lifelong Interaction Value of User-Author pairs (RLIV-UA) based on each interaction of UA pairs. To address the long intervals between UA interactions and the large scale of the UA space, we propose a novel Sparse Cross-Request Interaction Markov Decision Process (SCRI-MDP) and introduce an Adjacent State Approximation (ASA) method to construct RL training samples. Additionally, we introduce Multi-Task Critic Learning (MTCL) to capture the progressive nature of UA interactions (click -> follow -> gift), where denser interaction signals are leveraged to compensate for the learning of sparse labels. Finally, an auxiliary supervised learning task is designed to enhance the convergence of the RLIV-UA model. In offline experiments and online A/B tests, the RLIV-UA model achieves both higher user satisfaction and higher platform profits than compared methods.