Misspecified $Q$-Learning with Sparse Linear Function Approximation: Tight Bounds on Approximation Error

TL;DR

This paper establishes tight bounds on the sample complexity for misspecified sparse linear Q-learning, demonstrating that polynomial sample complexity is achievable under certain sparsity and misspecification conditions, unlike the non-sparse case.

Contribution

The paper introduces a novel elimination-based algorithm for sparse linear Q-learning and proves tight upper and lower bounds on its sample complexity.

Findings

Achieves $O(H\,\epsilon)$-optimal policy with polynomial sample complexity in feature dimension and horizon.

Standard Bellman backup approaches are suboptimal for this problem.

Provides a matching lower bound, completing the theoretical understanding of the problem.

Abstract

The recent work by Dong & Yang (2023) showed for misspecified sparse linear bandits, one can obtain an $O\left(\epsilon\right)$-optimal policy using a polynomial number of samples when the sparsity is a constant, where $\epsilon$ is the misspecification error. This result is in sharp contrast to misspecified linear bandits without sparsity, which require an exponential number of samples to get the same guarantee. In order to study whether the analog result is possible in the reinforcement learning setting, we consider the following problem: assuming the optimal $Q$-function is a $d$-dimensional linear function with sparsity $k$ and misspecification error $\epsilon$, whether we can obtain an $O\left(\epsilon\right)$-optimal policy using number of samples polynomially in the feature dimension $d$. We first demonstrate why the standard approach based on Bellman backup or the existing optimistic value function elimination approach such as OLIVE (Jiang et al., 2017) achieves suboptimal guarantees for this problem. We then design a novel elimination-based algorithm to show one can obtain an $O\left(H\epsilon\right)$-optimal policy with sample complexity polynomially in the feature dimension $d$ and planning horizon $H$. Lastly, we complement our upper bound with an $\widetilde{\Omega}\left(H\epsilon\right)$ suboptimality lower bound, giving a complete picture of this problem.