Sample Complexity for Markov Decision Processes and Stochastic Optimal Control with Static Risk Measures

TL;DR

This paper introduces a state augmentation technique for static risk measures in Markov decision processes and stochastic control, enabling derivation of dynamic programming equations and analysis of sample complexities.

Contribution

It proposes a novel elementary state augmentation method for static risk measures, facilitating analysis of sample complexities in MDPs and stochastic control.

Findings

Derived dynamic programming equations on augmented space.

Analyzed sample complexities for finite and infinite horizon cases.

Applied approach to distributionally robust functionals including CVaR.

Abstract

We present an elementary state augmentation method for a class of static risk measure applied to the total cost for both Markov decision processes and stochastic optimal control, such that dynamic programming equations can be derived on the augmented space. Through this we discuss the sample complexities of these two problems for both finite-horizon and infinite-horizon settings. We demonstrate the application of the proposed approach through studying distributionally robust functional generated by $\phi$-divergences including conditional value-at-risk.