Enabling Pareto-Stationarity Exploration in Multi-Objective Reinforcement Learning: A Multi-Objective Weighted-Chebyshev Actor-Critic Approach

TL;DR

This paper introduces MOCHA, a novel algorithm for multi-objective reinforcement learning that systematically explores Pareto-stationary solutions with finite-time sample complexity guarantees, demonstrated to outperform baselines in simulations.

Contribution

The paper proposes MOCHA, integrating weighted-Chebychev and actor-critic methods, to explore Pareto-stationary solutions in MORL with theoretical sample complexity guarantees.

Findings

MOCHA achieves $ ilde{O}(rac{1}{ extepsilon^2})$ sample complexity.

MOCHA outperforms baseline MORL algorithms in simulations.

Sample complexity depends on the minimum weight entry $p_{min}$.

Abstract

In many multi-objective reinforcement learning (MORL) applications, being able to systematically explore the Pareto-stationary solutions under multiple non-convex reward objectives with theoretical finite-time sample complexity guarantee is an important and yet under-explored problem. This motivates us to take the first step and fill the important gap in MORL. Specifically, in this paper, we propose a \uline{M}ulti-\uline{O}bjective weighted-\uline{CH}ebyshev \uline{A}ctor-critic (MOCHA) algorithm for MORL, which judiciously integrates the weighted-Chebychev (WC) and actor-critic framework to enable Pareto-stationarity exploration systematically with finite-time sample complexity guarantee. Sample complexity result of MOCHA algorithm reveals an interesting dependency on $p_{\min}$ in finding an $\epsilon$-Pareto-stationary solution, where $p_{\min}$ denotes the minimum entry of a given weight vector $\mathbf{p}$ in WC-scarlarization. By carefully choosing learning rates, the sample complexity for each exploration can be $\tilde{\mathcal{O}}(\epsilon^{-2})$. Furthermore, simulation studies on a large KuaiRand offline dataset, show that the performance of MOCHA algorithm significantly outperforms other baseline MORL approaches.