On the Convergence of Modified Policy Iteration in Risk Sensitive Exponential Cost Markov Decision Processes

TL;DR

This paper proves the convergence of modified policy iteration for risk-sensitive exponential cost MDPs with finite states and actions, and demonstrates its computational advantages over traditional methods.

Contribution

First convergence proof of MPI for risk-sensitive exponential cost MDPs, extending dynamic programming theory to this robust formulation.

Findings

MPI converges for risk-sensitive exponential cost MDPs.

MPI outperforms value and policy iteration in computational efficiency.

Simulation results confirm MPI's effectiveness across various parameters.

Abstract

Modified policy iteration (MPI) is a dynamic programming algorithm that combines elements of policy iteration and value iteration. The convergence of MPI has been well studied in the context of discounted and average-cost MDPs. In this work, we consider the exponential cost risk-sensitive MDP formulation, which is known to provide some robustness to model parameters. Although policy iteration and value iteration have been well studied in the context of risk sensitive MDPs, MPI is unexplored. We provide the first proof that MPI also converges for the risk-sensitive problem in the case of finite state and action spaces. Since the exponential cost formulation deals with the multiplicative Bellman equation, our main contribution is a convergence proof which is quite different than existing results for discounted and risk-neutral average-cost problems as well as risk sensitive value and policy iteration approaches. We conclude our analysis with simulation results, assessing MPI's performance relative to alternative dynamic programming methods like value iteration and policy iteration across diverse problem parameters. Our findings highlight risk-sensitive MPI's enhanced computational efficiency compared to both value and policy iteration techniques.