Necessary and Sufficient Oracles: Toward a Computational Taxonomy For Reinforcement Learning

TL;DR

This paper develops a computational taxonomy for reinforcement learning by analyzing the impact of different supervised learning oracles on RL complexity, identifying minimal and near-minimal oracles in various models.

Contribution

It introduces a formal framework to compare supervised learning oracles in RL, establishing minimality results and the benefits of resets in certain settings.

Findings

Two-context regression is minimal in Block MDPs with episodic access.

One-context regression is near-minimal with reset access, showing reset benefits.

Cryptographic evidence shows the minimal oracle is insufficient for Low-Rank MDPs.

Abstract

Algorithms for reinforcement learning (RL) in large state spaces crucially rely on supervised learning subroutines to estimate objects such as value functions or transition probabilities. Since only the simplest supervised learning problems can be solved provably and efficiently, practical performance of an RL algorithm depends on which of these supervised learning "oracles" it assumes access to (and how they are implemented). But which oracles are better or worse? Is there a minimal oracle? In this work, we clarify the impact of the choice of supervised learning oracle on the computational complexity of RL, as quantified by the oracle strength. First, for the task of reward-free exploration in Block MDPs in the standard episodic access model -- a ubiquitous setting for RL with function approximation -- we identify two-context regression as a minimal oracle, i.e. an oracle that is both necessary and sufficient (under a mild regularity assumption). Second, we identify one-context regression as a near-minimal oracle in the stronger reset access model, establishing a provable computational benefit of resets in the process. Third, we broaden our focus to Low-Rank MDPs, where we give cryptographic evidence that the analogous oracle from the Block MDP setting is insufficient.