Scalable Multi-Objective Reinforcement Learning with Fairness Guarantees using Lorenz Dominance

TL;DR

This paper presents a scalable multi-objective reinforcement learning algorithm that incorporates fairness using Lorenz dominance, enabling equitable policy discovery in complex, many-objective environments like large-scale transport planning.

Contribution

It introduces a novel Lorenz dominance-based approach with lambda-Lorenz dominance for flexible fairness, improving scalability and fairness in many-objective reinforcement learning.

Findings

Outperforms existing methods in high-dimensional objective spaces

Encourages discovery of fair policies in large-scale environments

Demonstrates scalability in real-world transport planning scenarios

Abstract

Multi-Objective Reinforcement Learning (MORL) aims to learn a set of policies that optimize trade-offs between multiple, often conflicting objectives. MORL is computationally more complex than single-objective RL, particularly as the number of objectives increases. Additionally, when objectives involve the preferences of agents or groups, incorporating fairness becomes both important and socially desirable. This paper introduces a principled algorithm that incorporates fairness into MORL while improving scalability to many-objective problems. We propose using Lorenz dominance to identify policies with equitable reward distributions and introduce lambda-Lorenz dominance to enable flexible fairness preferences. We release a new, large-scale real-world transport planning environment and demonstrate that our method encourages the discovery of fair policies, showing improved scalability in two large cities (Xi'an and Amsterdam). Our methods outperform common multi-objective approaches, particularly in high-dimensional objective spaces.