A Scale-Independent Multi-Objective Reinforcement Learning with Convergence Analysis

TL;DR

This paper introduces a scale-independent multi-objective reinforcement learning algorithm based on A2C, with proven convergence properties, that outperforms single-objective methods in multi-task experiments.

Contribution

It proposes a novel scale-independent multi-objective RL algorithm with convergence analysis, addressing issues of conflicting objectives with different scales.

Findings

The proposed algorithm demonstrates superior performance over single-objective methods.

Convergence-in-mean guarantee is established for the algorithm.

Experimental results validate the effectiveness of the approach.

Abstract

Many sequential decision-making problems need optimization of different objectives which possibly conflict with each other. The conventional way to deal with a multi-task problem is to establish a scalar objective function based on a linear combination of different objectives. However, for the case of having conflicting objectives with different scales, this method needs a trial-and-error approach to properly find proper weights for the combination. As such, in most cases, this approach cannot guarantee an optimal Pareto solution. In this paper, we develop a single-agent scale-independent multi-objective reinforcement learning on the basis of the Advantage Actor-Critic (A2C) algorithm. A convergence analysis is then done for the devised multi-objective algorithm providing a convergence-in-mean guarantee. We then perform some experiments over a multi-task problem to evaluate the performance of the proposed algorithm. Simulation results show the superiority of developed multi-objective A2C approach against the single-objective algorithm.