Pseudo-Quantized Actor-Critic Algorithm for Robustness to Noisy Temporal Difference Error

TL;DR

This paper introduces a novel reinforcement learning algorithm that enhances robustness to noisy temporal difference errors by pseudo-quantizing TD errors, leading to more stable learning without heavy heuristics.

Contribution

It proposes a control-as-inference based algorithm that decomposes optimality, enabling pseudo-quantization of TD errors for improved noise robustness in RL.

Findings

Demonstrates stable learning in benchmarks with noisy rewards.

Reduces reliance on heuristics like target networks and ensembles.

Achieves robustness without increased computational cost.

Abstract

In reinforcement learning (RL), temporal difference (TD) errors are widely adopted for optimizing value and policy functions. However, since the TD error is defined by a bootstrap method, its computation tends to be noisy and destabilize learning. Heuristics to improve the accuracy of TD errors, such as target networks and ensemble models, have been introduced so far. While these are essential approaches for the current deep RL algorithms, they cause side effects like increased computational cost and reduced learning efficiency. Therefore, this paper revisits the TD learning algorithm based on control as inference, deriving a novel algorithm capable of robust learning against noisy TD errors. First, the distribution model of optimality, a binary random variable, is represented by a sigmoid function. Alongside forward and reverse Kullback-Leibler divergences, this new model derives a robust learning rule: when the sigmoid function saturates with a large TD error probably due to noise, the gradient vanishes, implicitly excluding it from learning. Furthermore, the two divergences exhibit distinct gradient-vanishing characteristics. Building on these analyses, the optimality is decomposed into multiple levels to achieve pseudo-quantization of TD errors, aiming for further noise reduction. Additionally, a Jensen-Shannon divergence-based approach is approximately derived to inherit the characteristics of both divergences. These benefits are verified through RL benchmarks, demonstrating stable learning even when heuristics are insufficient or rewards contain noise.