Complexity scaling and optimal policy degeneracy in quantum reinforcement learning via analytically solvable unitary-control-then-measure models

TL;DR

This paper introduces analytically solvable quantum reinforcement learning models using a unitary-control-then-measure protocol, revealing complexity reductions and unique degeneracy phenomena in optimal policies.

Contribution

It provides explicit solutions for quantum RL models and uncovers structural complexity reductions and policy degeneracy phenomena not seen in measurement-free control.

Findings

Expected return complexity reduces from exponential to polynomial in trajectory length.

Identifies two levels of complexity reduction: trajectory-based and policy-based.

Discovers unique degeneracy behaviors of optimal policies influenced by quantum Zeno effect.

Abstract

We propose and analyse a class of analytically solvable models of quantum reinforcement learning (QRL), formulated as finite-horizon Markov decision processes in finite-dimensional Hilbert spaces. The models are built around a `unitary-control-then-measure' protocol, in which a learning agent applies unitary transformations to a quantum state and interleaves each control step with a projective measurement onto a prescribed reference basis. Exact closed-form expressions for trajectory probabilities, rewards, and the expected return are derived for four concrete realisations: a closed-chain and an anti-periodic qubit implementation, a qutrit model with ladder coupling, and a four-level two-qubit system. Two structural features of these QRL protocols are rigorously analysed. First, we identify and quantify a two-level reduction in the computational complexity of the expected return, from the nominally exponential $O(e^N)$ scaling in the trajectory length~$N$ to an explicit power-law $O(N^{\mathcal{I}})$: a trajectory-based level, arising from equivalence classes of paths sharing the same unordered state counts and transition frequencies, and a policy-based level, arising from the sparsity of the transition graph enforced by constrained unitary actions. Second, we characterise the degeneracy of optimal policies. The low-dimensional models exhibit unique optima whose asymptotic behaviour with~$N$ is governed by the quantum Zeno effect, while the four-level system displays both plateau-type quasi-degeneracy at large horizons and genuine discrete degeneracy at critical energy parameters -- phenomena with no counterpart in the measurement-free quantum optimal control landscape.