Multi-objective Reinforcement Learning With Augmented States Requires Rewards After Deployment

TL;DR

This paper highlights that multi-objective reinforcement learning with augmented states necessitates ongoing access to reward signals after deployment, impacting practical implementation and deployment strategies.

Contribution

It reveals a previously unnoticed requirement that MORL with augmented states needs reward signals post-deployment, affecting real-world application feasibility.

Findings

Augmented states in MORL require reward signals after deployment.

This requirement impacts the practicality of deploying MORL agents.

The paper clarifies a key distinction between MORL and single-objective RL.

Abstract

This research note identifies a previously overlooked distinction between multi-objective reinforcement learning (MORL), and more conventional single-objective reinforcement learning (RL). It has previously been noted that the optimal policy for an MORL agent with a non-linear utility function is required to be conditioned on both the current environmental state and on some measure of the previously accrued reward. This is generally implemented by concatenating the observed state of the environment with the discounted sum of previous rewards to create an augmented state. While augmented states have been widely-used in the MORL literature, one implication of their use has not previously been reported -- namely that they require the agent to have continued access to the reward signal (or a proxy thereof) after deployment, even if no further learning is required. This note explains why this is the case, and considers the practical repercussions of this requirement.