Provably Robust Temporal Difference Learning for Heavy-Tailed Rewards

TL;DR

This paper introduces a robust temporal difference learning method with dynamic gradient clipping to handle heavy-tailed reward distributions in reinforcement learning, providing provable guarantees and improved sample complexity.

Contribution

It develops a provably robust TD learning algorithm with dynamic gradient clipping for heavy-tailed rewards, improving theoretical guarantees over existing methods.

Findings

Achieves sample complexity of order O(ε^{-1/p}) with heavy-tailed rewards.

Provides high-probability bounds for the robust TD learning.

Numerical experiments validate the theoretical results.

Abstract

In a broad class of reinforcement learning applications, stochastic rewards have heavy-tailed distributions, which lead to infinite second-order moments for stochastic (semi)gradients in policy evaluation and direct policy optimization. In such instances, the existing RL methods may fail miserably due to frequent statistical outliers. In this work, we establish that temporal difference (TD) learning with a dynamic gradient clipping mechanism, and correspondingly operated natural actor-critic (NAC), can be provably robustified against heavy-tailed reward distributions. It is shown in the framework of linear function approximation that a favorable tradeoff between bias and variability of the stochastic gradients can be achieved with this dynamic gradient clipping mechanism. In particular, we prove that robust versions of TD learning achieve sample complexities of order $\mathcal{O}(\varepsilon^{-\frac{1}{p}})$ and $\mathcal{O}(\varepsilon^{-1-\frac{1}{p}})$ with and without the full-rank assumption on the feature matrix, respectively, under heavy-tailed rewards with finite moments of order $(1+p)$ for some $p\in(0,1]$, both in expectation and with high probability. We show that a robust variant of NAC based on Robust TD learning achieves $\tilde{\mathcal{O}}(\varepsilon^{-4-\frac{2}{p}})$ sample complexity. We corroborate our theoretical results with numerical experiments.