Acceleration Operators in the Value Iteration Algorithms for Average Reward Markov Decision Processes

TL;DR

This paper introduces acceleration operators to enhance the efficiency of value iteration algorithms for average reward Markov Decision Processes, addressing computational challenges in solving large-scale problems.

Contribution

It proposes novel acceleration operators based on contraction and monotonicity properties to improve value iteration performance for average reward MDPs.

Findings

Acceleration operators significantly speed up convergence.

Enhanced algorithms handle larger MDPs more efficiently.

Combines acceleration with stochastic shortest path methods.

Abstract

One of the most widely used methods for solving average cost MDP problems is the value iteration method. This method, however, is often computationally impractical and restricted in size of solvable MDP problems. We propose acceleration operators that improve the performance of the value iteration for average reward MDP models. These operators are based on two important properties of Markovian operator: contraction mapping and monotonicity. It is well known that the classical relative value iteration methods for average cost criteria MDP do not involve the max-norm contraction or monotonicity property. To overcome this difficulty we propose to combine acceleration operators with variants of value iteration for stochastic shortest path problems associated average reward problems.