Block Policy Mirror Descent

TL;DR

This paper introduces the block policy mirror descent (BPMD) method for reinforcement learning, offering efficient, convergent policy updates with theoretical guarantees and extensions to stochastic settings, advancing large-scale RL solutions.

Contribution

The paper develops and analyzes the first block coordinate descent methods for policy optimization in reinforcement learning, providing convergence guarantees and efficiency improvements.

Findings

BPMD achieves fast linear convergence to the global optimum.

Uniform sampling yields comparable complexity to batch methods.

Hybrid sampling schemes can accelerate convergence depending on the problem instance.

Abstract

In this paper, we present a new policy gradient (PG) methods, namely the block policy mirror descent (BPMD) method for solving a class of regularized reinforcement learning (RL) problems with (strongly)-convex regularizers. Compared to the traditional PG methods with a batch update rule, which visits and updates the policy for every state, BPMD method has cheap per-iteration computation via a partial update rule that performs the policy update on a sampled state. Despite the nonconvex nature of the problem and a partial update rule, we provide a unified analysis for several sampling schemes, and show that BPMD achieves fast linear convergence to the global optimality. In particular, uniform sampling leads to comparable worst-case total computational complexity as batch PG methods. A necessary and sufficient condition for convergence with on-policy sampling is also identified. With a hybrid sampling scheme, we further show that BPMD enjoys potential instance-dependent acceleration, leading to improved dependence on the state space and consequently outperforming batch PG methods. We then extend BPMD methods to the stochastic setting, by utilizing stochastic first-order information constructed from samples. With a generative model, $\tilde{\mathcal{O}}(\left\lvert \mathcal{S}\right\rvert \left\lvert \mathcal{A}\right\rvert /\epsilon)$ (resp. $\tilde{\mathcal{O}}(\left\lvert \mathcal{S}\right\rvert \left\lvert \mathcal{A} \right\rvert /\epsilon^2)$) sample complexities are established for the strongly-convex (resp. non-strongly-convex) regularizers, where $\epsilon$ denotes the target accuracy. To the best of our knowledge, this is the first time that block coordinate descent methods have been developed and analyzed for policy optimization in reinforcement learning, which provides a new perspective on solving large-scale RL problems.