Adversarial Online Multi-Task Reinforcement Learning

TL;DR

This paper studies adversarial online multi-task reinforcement learning, establishing fundamental lower bounds and proposing an algorithm that nearly matches these bounds, advancing understanding of task separation and sample efficiency.

Contribution

The paper introduces a new $ extit{2-JAO}$ MDP construction, derives tight lower bounds on regret and sample complexity, and presents a polynomial-time algorithm with near-optimal guarantees.

Findings

Lower bound of $oldsymbol{ extit{ extOmega}}(K extsqrt{DSAH})$ on regret.

Instance-specific lower bound of $oldsymbol{ extit{ extOmega}}(rac{K}{ extlambda^2})$ on sample complexity.

Proposed algorithm achieves near-optimal sample and regret bounds.

Abstract

We consider the adversarial online multi-task reinforcement learning setting, where in each of $K$ episodes the learner is given an unknown task taken from a finite set of $M$ unknown finite-horizon MDP models. The learner's objective is to minimize its regret with respect to the optimal policy for each task. We assume the MDPs in $\mathcal{M}$ are well-separated under a notion of $\lambda$-separability, and show that this notion generalizes many task-separability notions from previous works. We prove a minimax lower bound of $\Omega(K\sqrt{DSAH})$ on the regret of any learning algorithm and an instance-specific lower bound of $\Omega(\frac{K}{\lambda^2})$ in sample complexity for a class of uniformly-good cluster-then-learn algorithms. We use a novel construction called 2-JAO MDP for proving the instance-specific lower bound. The lower bounds are complemented with a polynomial time algorithm that obtains $\tilde{O}(\frac{K}{\lambda^2})$ sample complexity guarantee for the clustering phase and $\tilde{O}(\sqrt{MK})$ regret guarantee for the learning phase, indicating that the dependency on $K$ and $\frac{1}{\lambda^2}$ is tight.