Model Selection in Reinforcement Learning with General Function Approximations

TL;DR

This paper develops adaptive model selection algorithms for reinforcement learning environments like MABs and MDPs, which identify the correct function class among nested options, achieving near-oracle regret bounds with minimal additional cost.

Contribution

It introduces efficient algorithms for model selection in RL that adaptively find the smallest true function class, matching oracle performance under certain assumptions.

Findings

Regret bounds match oracle with known true class

Additive regret cost with logarithmic dependence on horizon

Algorithms adapt to the smallest true model class

Abstract

We consider model selection for classic Reinforcement Learning (RL) environments -- Multi Armed Bandits (MABs) and Markov Decision Processes (MDPs) -- under general function approximations. In the model selection framework, we do not know the function classes, denoted by $\mathcal{F}$ and $\mathcal{M}$, where the true models -- reward generating function for MABs and and transition kernel for MDPs -- lie, respectively. Instead, we are given $M$ nested function (hypothesis) classes such that true models are contained in at-least one such class. In this paper, we propose and analyze efficient model selection algorithms for MABs and MDPs, that \emph{adapt} to the smallest function class (among the nested $M$ classes) containing the true underlying model. Under a separability assumption on the nested hypothesis classes, we show that the cumulative regret of our adaptive algorithms match to that of an oracle which knows the correct function classes (i.e., $\cF$ and $\cM$) a priori. Furthermore, for both the settings, we show that the cost of model selection is an additive term in the regret having weak (logarithmic) dependence on the learning horizon $T$.