Provably efficient RL with Rich Observations via Latent State Decoding

TL;DR

This paper introduces a method for efficient reinforcement learning in environments with rich observations by decoding latent states, providing theoretical guarantees and empirical improvements over traditional Q-learning.

Contribution

It proposes a novel latent state decoding approach with finite-sample guarantees, significantly enhancing exploration efficiency in complex MDPs.

Findings

Method exponentially outperforms naive Q-learning in exploration tasks.

Finite-sample guarantees for decoding and policy quality.

Empirical validation on challenging exploration problems.

Abstract

We study the exploration problem in episodic MDPs with rich observations generated from a small number of latent states. Under certain identifiability assumptions, we demonstrate how to estimate a mapping from the observations to latent states inductively through a sequence of regression and clustering steps -- where previously decoded latent states provide labels for later regression problems -- and use it to construct good exploration policies. We provide finite-sample guarantees on the quality of the learned state decoding function and exploration policies, and complement our theory with an empirical evaluation on a class of hard exploration problems. Our method exponentially improves over $Q$-learning with na\"ive exploration, even when $Q$-learning has cheating access to latent states.