A nearly Blackwell-optimal policy gradient method

TL;DR

This paper introduces a policy gradient method that optimizes both the long-term gain and the transient bias in reinforcement learning, providing a more comprehensive approach to policy evaluation.

Contribution

It develops a novel bi-level optimization algorithm that jointly considers gain and bias, incorporating an RL-specific logarithmic barrier for improved policy selection.

Findings

Effective bias optimization improves transient performance.

The method offers insights into gain and bias trade-offs.

Experimental results validate the approach's advantages.

Abstract

For continuing environments, reinforcement learning (RL) methods commonly maximize the discounted reward criterion with discount factor close to 1 in order to approximate the average reward (the gain). However, such a criterion only considers the long-run steady-state performance, ignoring the transient behaviour in transient states. In this work, we develop a policy gradient method that optimizes the gain, then the bias (which indicates the transient performance and is important to capably select from policies with equal gain). We derive expressions that enable sampling for the gradient of the bias and its preconditioning Fisher matrix. We further devise an algorithm that solves the gain-then-bias (bi-level) optimization. Its key ingredient is an RL-specific logarithmic barrier function. Experimental results provide insights into the fundamental mechanisms of our proposal.