VO$Q$L: Towards Optimal Regret in Model-free RL with Nonlinear Function Approximation

TL;DR

This paper introduces VO$Q$L, a new algorithm for model-free reinforcement learning with nonlinear function approximation, achieving near-optimal regret bounds and computational efficiency in linear MDPs.

Contribution

The paper proposes VO$Q$L, a novel, computationally efficient algorithm with provably optimal regret bounds for RL with nonlinear function approximation.

Findings

Achieves $ ilde{O}(d ext{sqrt}(HT)+d^6H^{5})$ regret in linear MDPs.

First computationally tractable and statistically optimal approach for linear MDPs.

Incorporates weighted regression bounds to improve regret performance.

Abstract

We study time-inhomogeneous episodic reinforcement learning (RL) under general function approximation and sparse rewards. We design a new algorithm, Variance-weighted Optimistic $Q$-Learning (VO$Q$L), based on $Q$-learning and bound its regret assuming completeness and bounded Eluder dimension for the regression function class. As a special case, VO$Q$L achieves $\tilde{O}(d\sqrt{HT}+d^6H^{5})$ regret over $T$ episodes for a horizon $H$ MDP under ($d$-dimensional) linear function approximation, which is asymptotically optimal. Our algorithm incorporates weighted regression-based upper and lower bounds on the optimal value function to obtain this improved regret. The algorithm is computationally efficient given a regression oracle over the function class, making this the first computationally tractable and statistically optimal approach for linear MDPs.