A General Framework for Sample-Efficient Function Approximation in Reinforcement Learning

TL;DR

This paper introduces a unified framework for efficient function approximation in reinforcement learning, combining model-based and model-free approaches, and proposes a new algorithm with improved sample complexity for various MDP models.

Contribution

It presents a general framework and a novel algorithm, OPERA, that achieves improved sample efficiency and regret bounds across diverse MDP settings.

Findings

OPERA achieves regret bounds matching or surpassing existing methods.

For low Witness rank MDPs, OPERA improves sample complexity by a factor of dH.

The framework unifies and extends analysis for multiple RL models.

Abstract

With the increasing need for handling large state and action spaces, general function approximation has become a key technique in reinforcement learning (RL). In this paper, we propose a general framework that unifies model-based and model-free RL, and an Admissible Bellman Characterization (ABC) class that subsumes nearly all Markov Decision Process (MDP) models in the literature for tractable RL. We propose a novel estimation function with decomposable structural properties for optimization-based exploration and the functional eluder dimension as a complexity measure of the ABC class. Under our framework, a new sample-efficient algorithm namely OPtimization-based ExploRation with Approximation (OPERA) is proposed, achieving regret bounds that match or improve over the best-known results for a variety of MDP models. In particular, for MDPs with low Witness rank, under a slightly stronger assumption, OPERA improves the state-of-the-art sample complexity results by a factor of $dH$. Our framework provides a generic interface to design and analyze new RL models and algorithms.