On the Convergence of Policy in Unregularized Policy Mirror Descent

TL;DR

This paper analyzes the convergence behavior of policy in unregularized policy mirror descent using generalized Bregman divergence, revealing conditions for finite-step convergence to optimal policies.

Contribution

It extends previous work by providing convergence rates for policies under generalized Bregman divergence, including classical Euclidean distance, in unregularized policy mirror descent.

Findings

Finite-step convergence to optimal policy with certain Bregman divergences

Convergence rates established for policies under generalized Bregman divergence

Extension of previous convergence results in policy mirror descent

Abstract

In this short note, we give the convergence analysis of the policy in the recent famous policy mirror descent (PMD). We mainly consider the unregularized setting following [11] with generalized Bregman divergence. The difference is that we directly give the convergence rates of policy under generalized Bregman divergence. Our results are inspired by the convergence of value function in previous works and are an extension study of policy mirror descent. Though some results have already appeared in previous work, we further discover a large body of Bregman divergences could give finite-step convergence to an optimal policy, such as the classical Euclidean distance.