Optimal Rates of Convergence for Entropy Regularization in Discounted Markov Decision Processes

TL;DR

This paper establishes that entropy regularization in discounted Markov decision processes leads to errors that decrease exponentially with inverse regularization strength, providing precise convergence rates and insights into the implicit bias of natural policy gradient methods.

Contribution

It offers the first exponential convergence rate analysis for entropy regularization in MDPs, matching upper and lower bounds, and extends the analysis to general convex potentials.

Findings

Error decreases exponentially with inverse regularization strength.

Natural policy gradient methods exhibit exponential decay of error over iterations.

Extended analysis to general convex potentials and their natural policy gradients.

Abstract

We study the error introduced by entropy regularization in infinite-horizon discrete discounted Markov decision processes. We show that this error decreases exponentially in the inverse regularization strength, both in a weighted KL-divergence and in value with a problem-specific exponent. This is in contrast to previously known estimates, of the order $O(\tau)$, where $\tau$ is the regularization strength. We provide a lower bound that matches our upper bound up to a polynomial term, thereby characterizing the exponential convergence rate for entropy regularization. Our proof relies on the observation that the solutions of entropy-regularized Markov decision processes solve a gradient flow of the unregularized reward with respect to a Riemannian metric common in natural policy gradient methods. This correspondence allows us to identify the limit of this gradient flow as the generalized maximum entropy optimal policy, thereby characterizing the implicit bias of this gradient flow, which corresponds to a time-continuous version of the natural policy gradient method. We use our improved error estimates to show that for entropy-regularized natural policy gradient methods, the overall error decays exponentially in the square root of the number of iterations, improving over existing sublinear guarantees. Finally, we extend our analysis to settings beyond the entropy. In particular, we characterize the implicit bias regarding general convex potentials and their resulting generalized natural policy gradients.