Multi-Objective Recommendation via Multivariate Policy Learning

TL;DR

This paper introduces a novel multivariate policy learning framework for multi-objective recommendation systems, optimizing scalarisation weights as actions to improve long-term user engagement and fairness.

Contribution

It extends policy learning to continuous multivariate actions, proposing a pessimistic lower bound approach with correction techniques for better optimization.

Findings

Effective in simulations, offline, and online experiments

Improves balancing multiple objectives in recommender systems

Enhances long-term user engagement and fairness

Abstract

Real-world recommender systems often need to balance multiple objectives when deciding which recommendations to present to users. These include behavioural signals (e.g. clicks, shares, dwell time), as well as broader objectives (e.g. diversity, fairness). Scalarisation methods are commonly used to handle this balancing task, where a weighted average of per-objective reward signals determines the final score used for ranking. Naturally, how these weights are computed exactly, is key to success for any online platform. We frame this as a decision-making task, where the scalarisation weights are actions taken to maximise an overall North Star reward (e.g. long-term user retention or growth). We extend existing policy learning methods to the continuous multivariate action domain, proposing to maximise a pessimistic lower bound on the North Star reward that the learnt policy will yield. Typical lower bounds based on normal approximations suffer from insufficient coverage, and we propose an efficient and effective policy-dependent correction for this. We provide guidance to design stochastic data collection policies, as well as highly sensitive reward signals. Empirical observations from simulations, offline and online experiments highlight the efficacy of our deployed approach.