Fixed Points of the Set-Based Bellman Operator

TL;DR

This paper introduces a set-based Bellman operator for MDPs with uncertain cost parameters, proving the existence of a fixed point using contraction mapping principles, which enhances understanding of robustness in decision processes.

Contribution

It defines a novel set-based Bellman operator for MDPs with parameter uncertainty and proves its fixed point existence, extending classical Bellman operator theory.

Findings

Fixed point of the set-based Bellman operator exists.

The operator is shown to be contractive on a complete metric space.

Provides theoretical foundation for robust MDP solutions.

Abstract

Motivated by uncertain parameters encountered in Markov decision processes (MDPs), we study the effect of parameter uncertainty on Bellman operator-based methods. Specifically, we consider a family of MDPs where the cost parameters are from a given compact set. We then define a Bellman operator acting on an input set of value functions to produce a new set of value functions as the output under all possible variations in the cost parameters. Finally we prove the existence of a fixed point of this set-based Bellman operator by showing that it is a contractive operator on a complete metric space.