Navigating the Social Welfare Frontier: Portfolios for Multi-objective Reinforcement Learning

TL;DR

This paper introduces algorithms for creating policy portfolios in multi-objective reinforcement learning that approximate optimal solutions across a range of social welfare functions, aiding decision-makers in complex preference landscapes.

Contribution

It proposes the concept of an $oldsymbol{ ext{ extit{α}}}$-approximate portfolio in RL, providing algorithms and theoretical guarantees for multi-objective policy optimization across various social welfare functions.

Findings

Algorithms effectively summarize policy spaces for different welfare functions.

Theoretical guarantees on approximation trade-offs.

Experimental validation on synthetic and real datasets.

Abstract

In many real-world applications of reinforcement learning (RL), deployed policies have varied impacts on different stakeholders, creating challenges in reaching consensus on how to effectively aggregate their preferences. Generalized $p$-means form a widely used class of social welfare functions for this purpose, with broad applications in fair resource allocation, AI alignment, and decision-making. This class includes well-known welfare functions such as Egalitarian, Nash, and Utilitarian welfare. However, selecting the appropriate social welfare function is challenging for decision-makers, as the structure and outcomes of optimal policies can be highly sensitive to the choice of $p$. To address this challenge, we study the concept of an $\alpha$-approximate portfolio in RL, a set of policies that are approximately optimal across the family of generalized $p$-means for all $p \in [-\infty, 1]$. We propose algorithms to compute such portfolios and provide theoretical guarantees on the trade-offs among approximation factor, portfolio size, and computational efficiency. Experimental results on synthetic and real-world datasets demonstrate the effectiveness of our approach in summarizing the policy space induced by varying $p$ values, empowering decision-makers to navigate this landscape more effectively.