Improved Worst-Case Regret Bounds for Randomized Least-Squares Value Iteration

TL;DR

This paper improves the theoretical worst-case regret bounds for a randomized value iteration algorithm in reinforcement learning, matching the best existing bounds and advancing understanding of regret minimization in finite-horizon MDPs.

Contribution

It introduces a clipping variant of RLSVI that achieves tighter worst-case regret bounds, aligning with state-of-the-art TS-based algorithms.

Findings

Achieves $ ilde{O}(H^2S\sqrt{AT})$ regret bound

Matches the best known worst-case regret bounds for RLSVI

Provides theoretical guarantees for a new clipped RLSVI algorithm

Abstract

This paper studies regret minimization with randomized value functions in reinforcement learning. In tabular finite-horizon Markov Decision Processes, we introduce a clipping variant of one classical Thompson Sampling (TS)-like algorithm, randomized least-squares value iteration (RLSVI). Our $\tilde{\mathrm{O}}(H^2S\sqrt{AT})$ high-probability worst-case regret bound improves the previous sharpest worst-case regret bounds for RLSVI and matches the existing state-of-the-art worst-case TS-based regret bounds.