Risk Aversion in Finite Markov Decision Processes Using Total Cost Criteria and Average Value at Risk

TL;DR

This paper introduces an algorithm for computing risk-averse policies in finite Markov Decision Processes using total cost and average value at risk, addressing the need for risk-sensitive decision making.

Contribution

It presents the first method combining AVaR with total cost criteria in MDPs, with conditions for efficient approximation and solution.

Findings

Risk-averse policies reduce probability of deadline violations

Algorithm provides cost distribution analysis

Method demonstrated in robot deployment scenario

Abstract

In this paper we present an algorithm to compute risk averse policies in Markov Decision Processes (MDP) when the total cost criterion is used together with the average value at risk (AVaR) metric. Risk averse policies are needed when large deviations from the expected behavior may have detrimental effects, and conventional MDP algorithms usually ignore this aspect. We provide conditions for the structure of the underlying MDP ensuring that approximations for the exact problem can be derived and solved efficiently. Our findings are novel inasmuch as average value at risk has not previously been considered in association with the total cost criterion. Our method is demonstrated in a rapid deployment scenario, whereby a robot is tasked with the objective of reaching a target location within a temporal deadline where increased speed is associated with increased probability of failure. We demonstrate that the proposed algorithm not only produces a risk averse policy reducing the probability of exceeding the expected temporal deadline, but also provides the statistical distribution of costs, thus offering a valuable analysis tool.