Provable Multi-Objective Reinforcement Learning with Generative Models

TL;DR

This paper introduces a model-based algorithm for multi-objective reinforcement learning that guarantees near-optimal policies with polynomial sample complexity, providing the first finite-sample analysis in this domain.

Contribution

It proposes the envelop value iteration algorithm for MORL, enabling efficient learning with finite data and offering the first finite-sample theoretical guarantees.

Findings

Algorithm achieves polynomial sample complexity.

Converges linearly to near-optimal value functions.

First finite-sample analysis for MORL algorithms.

Abstract

Multi-objective reinforcement learning (MORL) is an extension of ordinary, single-objective reinforcement learning (RL) that is applicable to many real-world tasks where multiple objectives exist without known relative costs. We study the problem of single policy MORL, which learns an optimal policy given the preference of objectives. Existing methods require strong assumptions such as exact knowledge of the multi-objective Markov decision process, and are analyzed in the limit of infinite data and time. We propose a new algorithm called model-based envelop value iteration (EVI), which generalizes the enveloped multi-objective $Q$-learning algorithm in Yang et al., 2019. Our method can learn a near-optimal value function with polynomial sample complexity and linear convergence speed. To the best of our knowledge, this is the first finite-sample analysis of MORL algorithms.