Verification of Markov Decision Processes with Risk-Sensitive Measures

TL;DR

This paper introduces a convex-concave programming approach to efficiently compute risk-sensitive policies in Markov decision processes with temporal logic constraints, using a measure from cumulative prospect theory.

Contribution

It presents a novel approximation method for nonlinear risk measures, enabling practical policy computation in complex MDPs with risk sensitivity.

Findings

Effective approximation of nonlinear risk measures

Efficient convex-concave programming implementation

Successful demonstration on multiple scenarios

Abstract

We develop a method for computing policies in Markov decision processes with risk-sensitive measures subject to temporal logic constraints. Specifically, we use a particular risk-sensitive measure from cumulative prospect theory, which has been previously adopted in psychology and economics. The nonlinear transformation of the probabilities and utility functions yields a nonlinear programming problem, which makes computation of optimal policies typically challenging. We show that this nonlinear weighting function can be accurately approximated by the difference of two convex functions. This observation enables efficient policy computation using convex-concave programming. We demonstrate the effectiveness of the approach on several scenarios.