Set-based value operators for non-stationary Markovian environments

TL;DR

This paper introduces set-based value operators for non-stationary Markov Decision Processes, providing a unified framework for robust and dynamic programming with uncertain parameters, and offers convergence guarantees and practical applications.

Contribution

It generalizes Bellman operators to set-based operators, weakens the rectangularity condition, and proves convergence and fixed point properties for uncertain and dynamic MDPs.

Findings

Set-based value operators are contractions on value function sets.

The containment condition generalizes the rectangularity condition.

Set convergence is proven for value iteration under parameter uncertainty.

Abstract

This paper analyzes finite state Markov Decision Processes (MDPs) with uncertain parameters in compact sets and re-examines results from robust MDP via set-based fixed point theory. To this end, we generalize the Bellman and policy evaluation operators to contracting operators on the value function space and denote them as \emph{value operators}. We lift these value operators to act on \emph{sets} of value functions and denote them as \emph{set-based value operators}. We prove that the set-based value operators are \emph{contractions} in the space of compact value function sets. Leveraging insights from set theory, we generalize the rectangularity condition in classic robust MDP literature to a containment condition for all value operators, which is weaker and can be applied to a larger set of parameter-uncertain MDPs and contracting operators in dynamic programming. We prove that both the rectangularity condition and the containment condition sufficiently ensure that the set-based value operator's fixed point set contains its own extrema elements. For convex and compact sets of uncertain MDP parameters, we show equivalence between the classic robust value function and the supremum of the fixed point set of the set-based Bellman operator. Under dynamically changing MDP parameters in compact sets, we prove a set convergence result for value iteration, which otherwise may not converge to a single value function. Finally, we derive novel guarantees for probabilistic path-planning problems in planet exploration and stratospheric station-keeping.