A Contracting Dynamical System Perspective toward Interval Markov Decision Processes

TL;DR

This paper introduces a dynamical systems perspective to Interval Markov Decision Processes with continuous actions, proposing new methods for efficient policy estimation and bounds using convex optimization and system theory.

Contribution

It demonstrates that value iterations in IMDPs are contracting dynamical systems and introduces an action-space relaxation IMDP for efficient computation and bounds.

Findings

Value iterations are monotone contracting dynamical systems.

Action-space relaxation provides an upper bound for the original IMDP.

Efficient policy optimization via convex optimization techniques.

Abstract

Interval Markov decision processes are a class of Markov models where the transition probabilities between the states belong to intervals. In this paper, we study the problem of efficient estimation of the optimal policies in Interval Markov Decision Processes (IMDPs) with continuous action-space. Given an IMDP, we show that the pessimistic (resp. the optimistic) value iterations, i.e., the value iterations under the assumption of a competitive adversary (resp. cooperative agent), are monotone dynamical systems and are contracting with respect to the $\ell_{\infty}$-norm. Inspired by this dynamical system viewpoint, we introduce another IMDP, called the action-space relaxation IMDP. We show that the action-space relaxation IMDP has two key features: (i) its optimal value is an upper bound for the optimal value of the original IMDP, and (ii) its value iterations can be efficiently solved using tools and techniques from convex optimization. We then consider the policy optimization problems at each step of the value iterations as a feedback controller of the value function. Using this system-theoretic perspective, we propose an iteration-distributed implementation of the value iterations for approximating the optimal value of the action-space relaxation IMDP.