Near-Optimal Regret Bounds for Multi-batch Reinforcement Learning

TL;DR

This paper introduces a near-optimal algorithm for multi-batch reinforcement learning in finite-horizon MDPs, achieving low regret with minimal batch updates, and establishes lower bounds on batch complexity.

Contribution

It presents the first near-optimal regret bound with logarithmic batch complexity and provides matching lower bounds on the number of batches needed.

Findings

Achieves $ ilde{O}( oot{2}{}SAH^3K)$ regret with $O(H+ ext{loglog}K)$ batches.

Establishes lower bounds on batch complexity for near-optimal regret.

Introduces efficient exploration strategies for unlearned states and transition models.

Abstract

In this paper, we study the episodic reinforcement learning (RL) problem modeled by finite-horizon Markov Decision Processes (MDPs) with constraint on the number of batches. The multi-batch reinforcement learning framework, where the agent is required to provide a time schedule to update policy before everything, which is particularly suitable for the scenarios where the agent suffers extensively from changing the policy adaptively. Given a finite-horizon MDP with $S$ states, $A$ actions and planning horizon $H$, we design a computational efficient algorithm to achieve near-optimal regret of $\tilde{O}(\sqrt{SAH^3K\ln(1/\delta)})$\footnote{$\tilde{O}(\cdot)$ hides logarithmic terms of $(S,A,H,K)$} in $K$ episodes using $O\left(H+\log_2\log_2(K) \right)$ batches with confidence parameter $\delta$. To our best of knowledge, it is the first $\tilde{O}(\sqrt{SAH^3K})$ regret bound with $O(H+\log_2\log_2(K))$ batch complexity. Meanwhile, we show that to achieve $\tilde{O}(\mathrm{poly}(S,A,H)\sqrt{K})$ regret, the number of batches is at least $\Omega\left(H/\log_A(K)+ \log_2\log_2(K) \right)$, which matches our upper bound up to logarithmic terms. Our technical contribution are two-fold: 1) a near-optimal design scheme to explore over the unlearned states; 2) an computational efficient algorithm to explore certain directions with an approximated transition model.