Quantum Computing Provides Exponential Regret Improvement in Episodic Reinforcement Learning

TL;DR

This paper introduces a quantum reinforcement learning algorithm that significantly reduces regret, achieving exponential improvement over classical methods by leveraging quantum mean estimation techniques.

Contribution

The paper proposes a novel quantum UCB-based framework for episodic RL that attains exponential regret reduction compared to classical algorithms.

Findings

Quantum algorithm achieves regret of  in K episodes.

Quantum mean estimation provides quadratic sample efficiency.

Experiments demonstrate performance gains in RL environments.

Abstract

In this paper, we investigate the problem of \textit{episodic reinforcement learning} with quantum oracles for state evolution. To this end, we propose an \textit{Upper Confidence Bound} (UCB) based quantum algorithmic framework to facilitate learning of a finite-horizon MDP. Our quantum algorithm achieves an exponential improvement in regret as compared to the classical counterparts, achieving a regret of $\Tilde{\mathcal{O}}(1)$ as compared to $\Tilde{\mathcal{O}}(\sqrt{K})$ \footnote{$\Tilde{\mathcal{O}}(\cdot)$ hides logarithmic terms.}, $K$ being the number of training episodes. In order to achieve this advantage, we exploit efficient quantum mean estimation technique that provides quadratic improvement in the number of i.i.d. samples needed to estimate the mean of sub-Gaussian random variables as compared to classical mean estimation. This improvement is a key to the significant regret improvement in quantum reinforcement learning. We provide proof-of-concept experiments on various RL environments that in turn demonstrate performance gains of the proposed algorithmic framework.