CleanQRL: Lightweight Single-file Implementations of Quantum Reinforcement Learning Algorithms

TL;DR

CleanQRL is a lightweight library providing single-file implementations of quantum reinforcement learning algorithms, facilitating easier development, benchmarking, and transition from theory to practice in quantum ML research.

Contribution

It introduces a standardized, easy-to-use library for QRL algorithms, inspired by classical CleanRL, integrating distributed computing and hyperparameter tuning tools.

Findings

Enables quick adaptation of QRL algorithms for research and development

Facilitates benchmarking against classical RL methods

Supports practical application development in quantum reinforcement learning

Abstract

At the interception between quantum computing and machine learning, Quantum Reinforcement Learning (QRL) has emerged as a promising research field. Due to its novelty, a standardized and comprehensive collection for QRL algorithms has not yet been established. Researchers rely on numerous software stacks for classical Reinforcement Learning (RL) as well as on various quantum computing frameworks for the implementation of the quantum subroutines of their QRL algorithms. Inspired by the CleanRL library for classical RL algorithms, we present CleanQRL, a library that offers single-script implementations of many QRL algorithms. Our library provides clear and easy to understand scripts that researchers can quickly adapt to their own needs. Alongside ray tune for distributed computing and streamlined hyperparameter tuning, CleanQRL uses weights&biases to log important metrics, which facilitates benchmarking against other classical and quantum implementations. The CleanQRL library enables researchers to easily transition from theoretical considerations to practical applications.