Latent Variable Representation for Reinforcement Learning

TL;DR

This paper introduces a latent variable model-based approach for reinforcement learning that enhances sample efficiency through a new planning algorithm with UCB exploration, supported by theoretical analysis and empirical validation.

Contribution

It presents a novel latent variable representation for RL value functions, enabling efficient planning and exploration with theoretical guarantees and superior empirical performance.

Findings

Proposed a kernel embedding-based UCB planning algorithm.

Established sample complexity bounds for the approach.

Demonstrated improved performance on benchmark tasks.

Abstract

Deep latent variable models have achieved significant empirical successes in model-based reinforcement learning (RL) due to their expressiveness in modeling complex transition dynamics. On the other hand, it remains unclear theoretically and empirically how latent variable models may facilitate learning, planning, and exploration to improve the sample efficiency of RL. In this paper, we provide a representation view of the latent variable models for state-action value functions, which allows both tractable variational learning algorithm and effective implementation of the optimism/pessimism principle in the face of uncertainty for exploration. In particular, we propose a computationally efficient planning algorithm with UCB exploration by incorporating kernel embeddings of latent variable models. Theoretically, we establish the sample complexity of the proposed approach in the online and offline settings. Empirically, we demonstrate superior performance over current state-of-the-art algorithms across various benchmarks.