Is Plug-in Solver Sample-Efficient for Feature-based Reinforcement Learning?

TL;DR

This paper analyzes the sample complexity of model-based reinforcement learning with feature representations, proving that a plug-in solver approach can be sample-efficient under certain conditions, with complexity depending only on feature dimension.

Contribution

It establishes the minimax sample complexity bounds for feature-based RL using a plug-in solver, including cases with and without anchor-states.

Findings

Sample complexity is $O(K/(1-)^3 ^2)$ under the anchor-state assumption.

The approach is effective even without anchor-states, showing flexibility.

Complexity depends only on feature dimension, not on state or action space.

Abstract

It is believed that a model-based approach for reinforcement learning (RL) is the key to reduce sample complexity. However, the understanding of the sample optimality of model-based RL is still largely missing, even for the linear case. This work considers sample complexity of finding an $\epsilon$-optimal policy in a Markov decision process (MDP) that admits a linear additive feature representation, given only access to a generative model. We solve this problem via a plug-in solver approach, which builds an empirical model and plans in this empirical model via an arbitrary plug-in solver. We prove that under the anchor-state assumption, which implies implicit non-negativity in the feature space, the minimax sample complexity of finding an $\epsilon$-optimal policy in a $\gamma$-discounted MDP is $O(K/(1-\gamma)^3\epsilon^2)$, which only depends on the dimensionality $K$ of the feature space and has no dependence on the state or action space. We further extend our results to a relaxed setting where anchor-states may not exist and show that a plug-in approach can be sample efficient as well, providing a flexible approach to design model-based algorithms for RL.