Fast Value Tracking for Deep Reinforcement Learning

TL;DR

This paper introduces LKTD, a scalable sampling algorithm based on Kalman filtering and SGMCMC, enabling uncertainty quantification in deep reinforcement learning for more robust decision-making.

Contribution

It presents a novel, scalable sampling method for deep RL that quantifies uncertainty and converges to a stationary distribution, enhancing robustness.

Findings

LKTD efficiently samples from the posterior of neural network parameters.

The algorithm's convergence to a stationary distribution is theoretically proven.

Uncertainty quantification improves policy robustness during training.

Abstract

Reinforcement learning (RL) tackles sequential decision-making problems by creating agents that interacts with their environment. However, existing algorithms often view these problem as static, focusing on point estimates for model parameters to maximize expected rewards, neglecting the stochastic dynamics of agent-environment interactions and the critical role of uncertainty quantification. Our research leverages the Kalman filtering paradigm to introduce a novel and scalable sampling algorithm called Langevinized Kalman Temporal-Difference (LKTD) for deep reinforcement learning. This algorithm, grounded in Stochastic Gradient Markov Chain Monte Carlo (SGMCMC), efficiently draws samples from the posterior distribution of deep neural network parameters. Under mild conditions, we prove that the posterior samples generated by the LKTD algorithm converge to a stationary distribution. This convergence not only enables us to quantify uncertainties associated with the value function and model parameters but also allows us to monitor these uncertainties during policy updates throughout the training phase. The LKTD algorithm paves the way for more robust and adaptable reinforcement learning approaches.