Benchmarking MOEAs for solving continuous multi-objective RL problems

TL;DR

This paper benchmarks various multi-objective evolutionary algorithms on complex multi-objective reinforcement learning problems, analyzing their effectiveness and the influence of problem features on algorithm performance.

Contribution

It introduces a framework for characterizing MORL instance complexity and evaluates MOEAs against scalarized formulations and quality indicators.

Findings

MOEAs show varying effectiveness depending on MORL problem features

Certain quality indicators like hypervolume are useful for MORL evaluation

Benchmarking reveals strengths and limitations of current MOEAs in MORL contexts

Abstract

Multi-objective reinforcement learning (MORL) addresses the challenge of simultaneously optimizing multiple, often conflicting, rewards, moving beyond the single-reward focus of conventional reinforcement learning (RL). This approach is essential for applications where agents must balance trade-offs between diverse goals, such as speed, energy efficiency, or stability, as a series of sequential decisions. This paper investigates the applicability and limitations of multi-objective evolutionary algorithms (MOEAs) in solving complex MORL problems. We assess whether these algorithms can effectively address the unique challenges posed by MORL and how MORL instances can serve as benchmarks to evaluate and improve MOEA performance. In particular, we propose a framework to characterize the features influencing MORL instance complexity, select representative MORL problems from the literature, and benchmark a suite of MOEAs alongside single-objective EAs using scalarized MORL formulations. Additionally, we evaluate the utility of existing multi-objective quality indicators in MORL scenarios, such as hypervolume conducting a comparison of the algorithms supported by statistical analysis. Our findings provide insights into the interplay between MORL problem characteristics and algorithmic effectiveness, highlighting opportunities for advancing both MORL research and the design of evolutionary algorithms.