Optimistic Policy Iteration for MDPs with Acyclic Transient State Structure

TL;DR

This paper proves the convergence of an optimistic policy iteration method for a special class of MDPs where the induced graph structure is acyclic after collapsing recurrent classes, enabling efficient policy evaluation.

Contribution

It introduces and analyzes an optimistic policy iteration algorithm tailored for MDPs with acyclic transient state structures, establishing its convergence.

Findings

Proves convergence of the proposed OPI method for the specified MDP class.

Identifies structural properties that facilitate policy iteration convergence.

Provides theoretical guarantees for a class of MDPs with acyclic graph structures.

Abstract

We consider Markov Decision Processes (MDPs) in which every stationary policy induces the same graph structure for the underlying Markov chain and further, the graph has the following property: if we replace each recurrent class by a node, then the resulting graph is acyclic. For such MDPs, we prove the convergence of the stochastic dynamics associated with a version of optimistic policy iteration (OPI), suggested in Tsitsiklis (2002), in which the values associated with all the nodes visited during each iteration of the OPI are updated.