EVAL: EigenVector-based Average-reward Learning

TL;DR

This paper extends entropy-regularized average-reward reinforcement learning to neural network function approximation, providing new theoretical insights and demonstrating improved stability and convergence on control benchmarks.

Contribution

It introduces a neural network-based approach for entropy-regularized average-reward RL, generalizing previous linear methods and enabling solutions without entropy regularization.

Findings

Method compares favorably in stability

Faster convergence on benchmarks

Theoretical link between objectives

Abstract

In reinforcement learning, two objective functions have been developed extensively in the literature: discounted and averaged rewards. The generalization to an entropy-regularized setting has led to improved robustness and exploration for both of these objectives. Recently, the entropy-regularized average-reward problem was addressed using tools from large deviation theory in the tabular setting. This method has the advantage of linearity, providing access to both the optimal policy and average reward-rate through properties of a single matrix. In this paper, we extend that framework to more general settings by developing approaches based on function approximation by neural networks. This formulation reveals new theoretical insights into the relationship between different objectives used in RL. Additionally, we combine our algorithm with a posterior policy iteration scheme, showing how our approach can also solve the average-reward RL problem without entropy-regularization. Using classic control benchmarks, we experimentally find that our method compares favorably with other algorithms in terms of stability and rate of convergence.