An Actor-Critic Algorithm with Function Approximation for Risk Sensitive Cost Markov Decision Processes

TL;DR

This paper introduces a model-free actor-critic algorithm with function approximation for risk-sensitive Markov decision processes, addressing the complexity of multiplicative cost structures and demonstrating superior performance through numerical experiments.

Contribution

It develops the first actor-critic algorithm tailored for risk-sensitive MDPs with exponentiated costs and provides convergence analysis and empirical validation.

Findings

Algorithm outperforms recent methods in numerical tests

Convergence of the proposed method is theoretically established

Effective handling of multiplicative cost structures in risk-sensitive settings

Abstract

In this paper, we consider the risk-sensitive cost criterion with exponentiated costs for Markov decision processes and develop a model-free policy gradient algorithm in this setting. Unlike additive cost criteria such as average or discounted cost, the risk-sensitive cost criterion is less studied due to the complexity resulting from the multiplicative structure of the resulting Bellman equation. We develop an actor-critic algorithm with function approximation in this setting and provide its asymptotic convergence analysis. We also show the results of numerical experiments that demonstrate the superiority in performance of our algorithm over other recent algorithms in the literature.