Deep Pareto Reinforcement Learning for Multi-Objective Recommender Systems

TL;DR

This paper introduces Deep Pareto Reinforcement Learning (DeepPRL), a novel method for multi-objective recommender systems that models complex relationships and personalizes recommendations, outperforming existing methods in offline and real-world tests.

Contribution

DeepPRL systematically models dynamic, heterogeneous relationships between multiple objectives and personalizes recommendations, improving both short-term and long-term performance.

Findings

Achieves significant Pareto-dominance over state-of-the-art baselines.

Improves three conflicting business objectives in Alibaba's video platform.

Demonstrates tangible economic benefits in real-world deployment.

Abstract

Optimizing multiple objectives simultaneously is an important task for recommendation platforms to improve their performance. However, this task is particularly challenging since the relationships between different objectives are heterogeneous across different consumers and dynamically fluctuating according to different contexts. Especially in those cases when objectives become conflicting with each other, the result of recommendations will form a pareto-frontier, where the improvements of any objective comes at the cost of a performance decrease of another objective. Existing multi-objective recommender systems do not systematically consider such dynamic relationships; instead, they balance between these objectives in a static and uniform manner, resulting in only suboptimal multi-objective recommendation performance. In this paper, we propose a Deep Pareto Reinforcement Learning (DeepPRL) approach, where we (1) comprehensively model the complex relationships between multiple objectives in recommendations; (2) effectively capture personalized and contextual consumer preference for each objective to provide better recommendations; (3) optimize both the short-term and the long-term performance of multi-objective recommendations. As a result, our method achieves significant pareto-dominance over the state-of-the-art baselines in the offline experiments. Furthermore, we conducted a controlled experiment at the video streaming platform of Alibaba, where our method simultaneously improved three conflicting business objectives over the latest production system significantly, demonstrating its tangible economic impact in practice.