Approximate Value Iteration for Risk-aware Markov Decision Processes

TL;DR

This paper introduces simulation-based algorithms for approximately solving large-scale risk-aware Markov decision processes, addressing the computational challenges posed by high-dimensional problems with risk considerations.

Contribution

It develops a family of approximate dynamic programming algorithms with convergence analysis and sample complexity bounds for large-scale risk-aware MDPs.

Findings

Algorithms effectively handle large-scale risk-aware MDPs.

Convergence and sample complexity bounds are established.

Method extends dynamic programming to high-dimensional risk-sensitive settings.

Abstract

We consider large-scale Markov decision processes (MDPs) with a risk measure of variability in cost, under the risk-aware MDPs paradigm. Previous studies showed that risk-aware MDPs, based on a minimax approach to handling risk, can be solved using dynamic programming for small to medium sized problems. However, due to the "curse of dimensionality", MDPs that model real-life problems are typically prohibitively large for such approaches. In this paper, we employ an approximate dynamic programming approach, and develop a family of simulation-based algorithms to approximately solve large-scale risk-aware MDPs. In parallel, we develop a unified convergence analysis technique to derive sample complexity bounds for this new family of algorithms.