A Demonstration of Issues with Value-Based Multiobjective Reinforcement Learning Under Stochastic State Transitions

TL;DR

This paper identifies a critical issue with value-based multiobjective reinforcement learning methods in stochastic environments, showing they can fail to find optimal policies and may converge to suboptimal solutions, and discusses better alternatives.

Contribution

It reveals a previously unknown problem with model-free, value-based methods in stochastic MOMDPs and suggests alternative approaches for maximizing Scalarised Expected Return.

Findings

Value-based methods may converge to Pareto-dominated solutions.

Standard approaches can fail to maximize Scalarised Expected Return in stochastic MOMDPs.

Alternative methods may better handle stochastic transitions in multiobjective RL.

Abstract

We report a previously unidentified issue with model-free, value-based approaches to multiobjective reinforcement learning in the context of environments with stochastic state transitions. An example multiobjective Markov Decision Process (MOMDP) is used to demonstrate that under such conditions these approaches may be unable to discover the policy which maximises the Scalarised Expected Return, and in fact may converge to a Pareto-dominated solution. We discuss several alternative methods which may be more suitable for maximising SER in MOMDPs with stochastic transitions.