VQC-Based Reinforcement Learning with Data Re-uploading: Performance and Trainability

TL;DR

This paper investigates the use of Variational Quantum Circuits in Deep Q-Learning, focusing on how data re-uploading influences performance and trainability in classic control tasks, revealing promising trainability properties.

Contribution

It provides an empirical analysis of VQC-based Deep Q-Learning, highlighting the effects of data re-uploading and the non-vanishing gradients with increasing qubits.

Findings

Gradients remain substantial throughout training.

Increasing qubits does not cause exponential gradient vanishing.

VQCs may be suitable as function approximators in RL.

Abstract

Reinforcement Learning (RL) consists of designing agents that make intelligent decisions without human supervision. When used alongside function approximators such as Neural Networks (NNs), RL is capable of solving extremely complex problems. Deep Q-Learning, a RL algorithm that uses Deep NNs, achieved super-human performance in some specific tasks. Nonetheless, it is also possible to use Variational Quantum Circuits (VQCs) as function approximators in RL algorithms. This work empirically studies the performance and trainability of such VQC-based Deep Q-Learning models in classic control benchmark environments. More specifically, we research how data re-uploading affects both these metrics. We show that the magnitude and the variance of the gradients of these models remain substantial throughout training due to the moving targets of Deep Q-Learning. Moreover, we empirically show that increasing the number of qubits does not lead to an exponential vanishing behavior of the magnitude and variance of the gradients for a PQC approximating a 2-design, unlike what was expected due to the Barren Plateau Phenomenon. This hints at the possibility of VQCs being specially adequate for being used as function approximators in such a context.