PAC Reinforcement Learning for Predictive State Representations

TL;DR

This paper introduces a novel model-based reinforcement learning algorithm for Predictive State Representations (PSRs) that achieves polynomial sample complexity, enabling near-optimal policy learning in partially observable systems with large state spaces.

Contribution

It develops the first polynomial sample complexity algorithm for PSRs that works with function approximation and applies to various POMDP subclasses.

Findings

Algorithm learns near-optimal policies with polynomial sample complexity.

Sample complexity scales polynomially with model class complexity, not state space size.

Applicable to multiple POMDP variants, including low-rank and linear models.

Abstract

In this paper we study online Reinforcement Learning (RL) in partially observable dynamical systems. We focus on the Predictive State Representations (PSRs) model, which is an expressive model that captures other well-known models such as Partially Observable Markov Decision Processes (POMDP). PSR represents the states using a set of predictions of future observations and is defined entirely using observable quantities. We develop a novel model-based algorithm for PSRs that can learn a near optimal policy in sample complexity scaling polynomially with respect to all the relevant parameters of the systems. Our algorithm naturally works with function approximation to extend to systems with potentially large state and observation spaces. We show that given a realizable model class, the sample complexity of learning the near optimal policy only scales polynomially with respect to the statistical complexity of the model class, without any explicit polynomial dependence on the size of the state and observation spaces. Notably, our work is the first work that shows polynomial sample complexities to compete with the globally optimal policy in PSRs. Finally, we demonstrate how our general theorem can be directly used to derive sample complexity bounds for special models including $m$-step weakly revealing and $m$-step decodable tabular POMDPs, POMDPs with low-rank latent transition, and POMDPs with linear emission and latent transition.