A Distributional View on Multi-Objective Policy Optimization

TL;DR

This paper introduces a scale-invariant multi-objective reinforcement learning algorithm that learns action distributions for each objective, enabling flexible preference setting and effective exploration of nondominated solutions in complex tasks.

Contribution

It proposes a novel method that allows setting preferences in a scale-invariant manner by learning distributions for each objective and fitting a combined policy.

Findings

Effective on high-dimensional robotics tasks

Enables tracing of nondominated solutions

Preference setting influences solution trade-offs

Abstract

Many real-world problems require trading off multiple competing objectives. However, these objectives are often in different units and/or scales, which can make it challenging for practitioners to express numerical preferences over objectives in their native units. In this paper we propose a novel algorithm for multi-objective reinforcement learning that enables setting desired preferences for objectives in a scale-invariant way. We propose to learn an action distribution for each objective, and we use supervised learning to fit a parametric policy to a combination of these distributions. We demonstrate the effectiveness of our approach on challenging high-dimensional real and simulated robotics tasks, and show that setting different preferences in our framework allows us to trace out the space of nondominated solutions.