Geometric Value Iteration: Dynamic Error-Aware KL Regularization for Reinforcement Learning

TL;DR

This paper introduces Geometric Value Iteration (GVI), a novel reinforcement learning algorithm that adaptively tunes KL regularization to improve robustness and stability, especially when using deep networks.

Contribution

It presents the first asymptotic error bound for dynamic KL coefficient schemes and proposes GVI, which dynamically adjusts regularization to balance learning speed and robustness.

Findings

GVI outperforms fixed-coefficient methods in stability and convergence.

Dynamic error-aware KL tuning improves robustness in deep RL.

GVI maintains stable learning without target networks.

Abstract

The recent boom in the literature on entropy-regularized reinforcement learning (RL) approaches reveals that Kullback-Leibler (KL) regularization brings advantages to RL algorithms by canceling out errors under mild assumptions. However, existing analyses focus on fixed regularization with a constant weighting coefficient and do not consider cases where the coefficient is allowed to change dynamically. In this paper, we study the dynamic coefficient scheme and present the first asymptotic error bound. Based on the dynamic coefficient error bound, we propose an effective scheme to tune the coefficient according to the magnitude of error in favor of more robust learning. Complementing this development, we propose a novel algorithm, Geometric Value Iteration (GVI), that features a dynamic error-aware KL coefficient design with the aim of mitigating the impact of errors on performance. Our experiments demonstrate that GVI can effectively exploit the trade-off between learning speed and robustness over uniform averaging of a constant KL coefficient. The combination of GVI and deep networks shows stable learning behavior even in the absence of a target network, where algorithms with a constant KL coefficient would greatly oscillate or even fail to converge.