A Study on Optimization Techniques for Variational Quantum Circuits in Reinforcement Learning

TL;DR

This paper explores optimization techniques for variational quantum circuits in reinforcement learning, demonstrating that data re-uploading and exponential learning rate decay improve performance and stability in noisy quantum environments.

Contribution

It introduces and evaluates specific optimization strategies, including data re-uploading and exponential learning rate decay, for VQCs in reinforcement learning tasks.

Findings

Data re-uploading enhances hyperparameter stability.

Exponential learning rate decay improves overall performance.

Output scaling increases learning speed and robustness.

Abstract

Quantum Computing aims to streamline machine learning, making it more effective with fewer trainable parameters. This reduction of parameters can speed up the learning process and reduce the use of computational resources. However, in the current phase of quantum computing development, known as the noisy intermediate-scale quantum era (NISQ), learning is difficult due to a limited number of qubits and widespread quantum noise. To overcome these challenges, researchers are focusing on variational quantum circuits (VQCs). VQCs are hybrid algorithms that merge a quantum circuit, which can be adjusted through parameters, with traditional classical optimization techniques. These circuits require only few qubits for effective learning. Recent studies have presented new ways of applying VQCs to reinforcement learning, showing promising results that warrant further exploration. This study investigates the effects of various techniques -- data re-uploading, input scaling, output scaling -- and introduces exponential learning rate decay in the quantum proximal policy optimization algorithm's actor-VQC. We assess these methods in the popular Frozen Lake and Cart Pole environments. Our focus is on their ability to reduce the number of parameters in the VQC without losing effectiveness. Our findings indicate that data re-uploading and an exponential learning rate decay significantly enhance hyperparameter stability and overall performance. While input scaling does not improve parameter efficiency, output scaling effectively manages greediness, leading to increased learning speed and robustness.