Performance of NPG in Countable State-Space Average-Cost RL

TL;DR

This paper analyzes the Natural Policy Gradient algorithm in countably infinite state-space reinforcement learning, proposing a state-dependent step-size rule that improves performance and achieves iteration complexity independent of state space size.

Contribution

It introduces a state-dependent step-size rule for NPG in large state spaces and provides theoretical bounds ensuring complexity independence from state space size.

Findings

NPG performance is significantly improved with the new step-size rule.

Iteration complexity of NPG can be made independent of state space size.

Theoretical bounds on Poisson's equation solutions support the step-size design.

Abstract

We consider policy optimization methods in reinforcement learning settings where the state space is arbitrarily large, or even countably infinite. The motivation arises from control problems in communication networks, matching markets, and other queueing systems. Specifically, we consider the popular Natural Policy Gradient (NPG) algorithm, which has been studied in the past only under the assumption that the cost is bounded and the state space is finite, neither of which holds for the aforementioned control problems. Assuming a Lyapunov drift condition, which is naturally satisfied in some cases and can be satisfied in other cases at a small cost in performance, we design a state-dependent step-size rule which dramatically improves the performance of NPG for our intended applications. In addition to experimentally verifying the performance improvement, we also theoretically show that the iteration complexity of NPG can be made independent of the size of the state space. The key analytical tool we use is the connection between NPG step-sizes and the solution to Poisson's equation. In particular, we provide policy-independent bounds on the solution to Poisson's equation, which are then used to guide the choice of NPG step-sizes.