Eluder-based Regret for Stochastic Contextual MDPs

TL;DR

This paper introduces the E-UC$^3$RL algorithm for regret minimization in stochastic contextual MDPs, leveraging Eluder dimension and offline regression oracles to achieve rate-optimal performance under minimal assumptions.

Contribution

It presents the first efficient, rate-optimal regret minimization algorithm for CMDPs using general offline function approximation and extends the Eluder dimension to bounded metrics.

Findings

Achieves regret bound of $ ilde{O}(H^3 oot{T |S| |A|} d_E(\

First efficient and rate-optimal algorithm for CMDPs with offline function approximation.

Extends Eluder dimension to general bounded metrics.

Abstract

We present the E-UC$^3$RL algorithm for regret minimization in Stochastic Contextual Markov Decision Processes (CMDPs). The algorithm operates under the minimal assumptions of realizable function class and access to \emph{offline} least squares and log loss regression oracles. Our algorithm is efficient (assuming efficient offline regression oracles) and enjoys a regret guarantee of $ \widetilde{O}(H^3 \sqrt{T |S| |A|d_{\mathrm{E}}(\mathcal{P}) \log (|\mathcal{F}| |\mathcal{P}|/ \delta) )}) , $ with $T$ being the number of episodes, $S$ the state space, $A$ the action space, $H$ the horizon, $\mathcal{P}$ and $\mathcal{F}$ are finite function classes used to approximate the context-dependent dynamics and rewards, respectively, and $d_{\mathrm{E}}(\mathcal{P})$ is the Eluder dimension of $\mathcal{P}$ w.r.t the Hellinger distance. To the best of our knowledge, our algorithm is the first efficient and rate-optimal regret minimization algorithm for CMDPs that operates under the general offline function approximation setting. In addition, we extend the Eluder dimension to general bounded metrics which may be of separate interest.