Quantum-enhanced reinforcement learning for finite-episode games with discrete state spaces

TL;DR

This paper demonstrates how quantum annealing can be used to enhance reinforcement learning in finite-episode games with discrete states by embedding policy evaluation and value function approximation as QUBO problems on a D-Wave quantum processor.

Contribution

It introduces methods to embed Monte Carlo policy evaluation and value function approximation into quantum annealing hardware for reinforcement learning tasks.

Findings

Quantum-enhanced policy evaluation finds better or equivalent value functions.

Embedding RL problems as QUBO enables quantum hardware utilization.

Quantum-classical algorithms improve efficiency in finite-episode games.

Abstract

Quantum annealing algorithms belong to the class of metaheuristic tools, applicable for solving binary optimization problems. Hardware implementations of quantum annealing, such as the quantum annealing machines produced by D-Wave Systems, have been subject to multiple analyses in research, with the aim of characterizing the technology's usefulness for optimization and sampling tasks. Here, we present a way to partially embed both Monte Carlo policy iteration for finding an optimal policy on random observations, as well as how to embed (n) sub-optimal state-value functions for approximating an improved state-value function given a policy for finite horizon games with discrete state spaces on a D-Wave 2000Q quantum processing unit (QPU). We explain how both problems can be expressed as a quadratic unconstrained binary optimization (QUBO) problem, and show that quantum-enhanced Monte Carlo policy evaluation allows for finding equivalent or better state-value functions for a given policy with the same number episodes compared to a purely classical Monte Carlo algorithm. Additionally, we describe a quantum-classical policy learning algorithm. Our first and foremost aim is to explain how to represent and solve parts of these problems with the help of the QPU, and not to prove supremacy over every existing classical policy evaluation algorithm.