On the Expressivity of Multidimensional Markov Reward

TL;DR

This paper explores the conditions under which multidimensional Markov reward functions can precisely characterize sets of desired policies in Markov Decision Processes, advancing understanding of reward design in sequential decision making.

Contribution

It provides necessary and sufficient conditions for the existence of reward functions that distinguish specific policy sets, including the construction of multidimensional rewards for deterministic policies.

Findings

Necessary and sufficient conditions for reward existence

Multidimensional rewards can characterize any non-degenerate deterministic policy set

Theoretical framework for reward design in MDPs

Abstract

We consider the expressivity of Markov rewards in sequential decision making under uncertainty. We view reward functions in Markov Decision Processes (MDPs) as a means to characterize desired behaviors of agents. Assuming desired behaviors are specified as a set of acceptable policies, we investigate if there exists a scalar or multidimensional Markov reward function that makes the policies in the set more desirable than the other policies. Our main result states both necessary and sufficient conditions for the existence of such reward functions. We also show that for every non-degenerate set of deterministic policies, there exists a multidimensional Markov reward function that characterizes it