Refined Policy Improvement Bounds for MDPs

TL;DR

This paper refines policy improvement bounds in Markov Decision Processes, providing a continuous bound in the discount factor that enhances the theoretical foundation of trust-region policy optimization, especially for high discount factors.

Contribution

The authors introduce a new policy improvement bound that remains valid as the discount factor approaches one, extending applicability to long-run average reward MDPs.

Findings

Bound is continuous in the discount factor

Applicable to long-run average reward MDPs

Improves theoretical justification of TRPO

Abstract

The policy improvement bound on the difference of the discounted returns plays a crucial role in the theoretical justification of the trust-region policy optimization (TRPO) algorithm. The existing bound leads to a degenerate bound when the discount factor approaches one, making the applicability of TRPO and related algorithms questionable when the discount factor is close to one. We refine the results in \cite{Schulman2015, Achiam2017} and propose a novel bound that is "continuous" in the discount factor. In particular, our bound is applicable for MDPs with the long-run average rewards as well.