Tight Performance Bounds for Approximate Modified Policy Iteration with Non-Stationary Policies

TL;DR

This paper demonstrates that non-stationary policies in approximate dynamic programming can yield significantly better performance guarantees than stationary ones, especially for high discount factors, by analyzing a unified non-stationary MPI algorithm.

Contribution

It introduces a non-stationary variant of Modified Policy Iteration that improves performance bounds and unifies existing approximate DP algorithms, with tight guarantees shown through a constructed example.

Findings

Performance bounds improved by a factor of O(1-γ)

Unification of value and policy iteration results

Tightness of the performance guarantee proven

Abstract

We consider approximate dynamic programming for the infinite-horizon stationary $\gamma$-discounted optimal control problem formalized by Markov Decision Processes. While in the exact case it is known that there always exists an optimal policy that is stationary, we show that when using value function approximation, looking for a non-stationary policy may lead to a better performance guarantee. We define a non-stationary variant of MPI that unifies a broad family of approximate DP algorithms of the literature. For this algorithm we provide an error propagation analysis in the form of a performance bound of the resulting policies that can improve the usual performance bound by a factor $O(1-\gamma)$, which is significant when the discount factor $\gamma$ is close to 1. Doing so, our approach unifies recent results for Value and Policy Iteration. Furthermore, we show, by constructing a specific deterministic MDP, that our performance guarantee is tight.