On Using Hamiltonian Monte Carlo Sampling for Reinforcement Learning Problems in High-dimension

TL;DR

This paper introduces Hamiltonian Q-Learning, a novel approach that leverages Hamiltonian Monte Carlo sampling and matrix completion to efficiently learn value functions in high-dimensional, stochastic reinforcement learning environments.

Contribution

It presents a new framework combining HMC sampling with matrix completion for effective Q-learning in complex high-dimensional stochastic settings.

Findings

Theoretically demonstrates the viability of HMC-generated data for Q-learning.

Empirically validates the approach on high-dimensional RL problems.

Shows improved data efficiency and scalability in complex environments.

Abstract

Value function based reinforcement learning (RL) algorithms, for example, $Q$-learning, learn optimal policies from datasets of actions, rewards, and state transitions. However, when the underlying state transition dynamics are stochastic and evolve on a high-dimensional space, generating independent and identically distributed (IID) data samples for creating these datasets poses a significant challenge due to the intractability of the associated normalizing integral. In these scenarios, Hamiltonian Monte Carlo (HMC) sampling offers a computationally tractable way to generate data for training RL algorithms. In this paper, we introduce a framework, called \textit{Hamiltonian $Q$-Learning}, that demonstrates, both theoretically and empirically, that $Q$ values can be learned from a dataset generated by HMC samples of actions, rewards, and state transitions. Furthermore, to exploit the underlying low-rank structure of the $Q$ function, Hamiltonian $Q$-Learning uses a matrix completion algorithm for reconstructing the updated $Q$ function from $Q$ value updates over a much smaller subset of state-action pairs. Thus, by providing an efficient way to apply $Q$-learning in stochastic, high-dimensional settings, the proposed approach broadens the scope of RL algorithms for real-world applications.