Quantum-Inspired Episode Selection for Monte Carlo Reinforcement Learning via QUBO Optimization

TL;DR

This paper introduces MC+QUBO, a quantum-inspired episode selection method for Monte Carlo reinforcement learning that improves convergence and policy quality by optimizing episode subsets with QUBO solvers.

Contribution

It reformulates episode selection as a QUBO problem and demonstrates the effectiveness of quantum-inspired samplers in enhancing reinforcement learning performance.

Findings

MC+QUBO outperforms vanilla Monte Carlo in convergence speed.

QUBO-based selection improves final policy quality.

Quantum-inspired optimization enhances decision-making in RL.

Abstract

Monte Carlo (MC) reinforcement learning suffers from high sample complexity, especially in environments with sparse rewards, large state spaces, and correlated trajectories. We address these limitations by reformulating episode selection as a Quadratic Unconstrained Binary Optimization (QUBO) problem and solving it with quantum-inspired samplers. Our method, MC+QUBO, integrates a combinatorial filtering step into standard MC policy evaluation: from each batch of trajectories, we select a subset that maximizes cumulative reward while promoting state-space coverage. This selection is encoded as a QUBO, where linear terms favor high-reward episodes and quadratic terms penalize redundancy. We explore both Simulated Quantum Annealing (SQA) and Simulated Bifurcation (SB) as black-box solvers within this framework. Experiments in a finite-horizon GridWorld demonstrate that MC+QUBO outperforms vanilla MC in convergence speed and final policy quality, highlighting the potential of quantum-inspired optimization as a decision-making subroutine in reinforcement learning.