On Oracle-Efficient PAC RL with Rich Observations

TL;DR

This paper develops new computationally efficient algorithms for PAC reinforcement learning with rich observations and deterministic hidden states, while highlighting fundamental challenges in more complex stochastic settings.

Contribution

Introduces oracle-efficient algorithms for PAC RL with rich observations and deterministic states, and analyzes limitations in stochastic hidden state environments.

Findings

New algorithms operate efficiently in oracle models

OLIVE cannot be implemented in the oracle model with stochastic dynamics

Fundamental challenges identified for PAC RL in complex settings

Abstract

We study the computational tractability of PAC reinforcement learning with rich observations. We present new provably sample-efficient algorithms for environments with deterministic hidden state dynamics and stochastic rich observations. These methods operate in an oracle model of computation -- accessing policy and value function classes exclusively through standard optimization primitives -- and therefore represent computationally efficient alternatives to prior algorithms that require enumeration. With stochastic hidden state dynamics, we prove that the only known sample-efficient algorithm, OLIVE, cannot be implemented in the oracle model. We also present several examples that illustrate fundamental challenges of tractable PAC reinforcement learning in such general settings.