Towards Real-time Control of a CartPole System on a Quantum Computer

TL;DR

This paper demonstrates a minimal hybrid quantum-classical approach to real-time CartPole control on a superconducting quantum processor, highlighting the impact of inference frequency and shot count on stability and performance.

Contribution

It presents the first end-to-end implementation of a quantum reinforcement learning agent on a physical QPU for real-time control, analyzing hardware latency and control trade-offs.

Findings

Single-qubit quantum agent outperforms classical in fewer episodes.

Higher inference frequency improves control stability.

Increased shot count reduces minimum frequency needed for near-maximal performance.

Abstract

The application of quantum reinforcement learning (QRL) to real-time control systems faces significant challenges regarding hardware latency, noise susceptibility, and learning convergence. This work presents an end-to-end investigation of a minimal hybrid quantum-classical agent applied to the CartPole benchmark, addressing the gap between idealized simulation and execution on a physical superconducting quantum processing unit (QPU). We demonstrate that a single-qubit agent acts as an effective learning model, solving the environment in substantially fewer episodes than a comparable classical actor-critic network even when the training of the hybrid agent is restricted to use parameter-shift for its quantum circuit component. To connect learning to deployment constraints, we map the inference-time trade-off between control-loop rate and measurement shot budget to provide guidance for an eventual real-time control demonstration. The resulting performance matrices show that both inference control frequency and shot count strongly affect balancing stability: higher inference frequencies consistently improve performance, and increasing the shot budget lowers the minimum inference frequency required to achieve near-maximal balancing. These results highlight the importance of finding an optimal medium between shot count and control frequency and developing circuits that are e.g. initial-state invariant. Lastly, we address the critical bottleneck of control latency on NISQ hardware by bypassing the standard high-level software stack and programming the Zurich Instruments readout electronics directly via command tables. These results quantify some of the current boundaries of quantum-assisted control and provide a start for achieving the tens-of-hertz throughput required for real-time closed-loop control feedback.