Controlled Online Optimization Learning (COOL): Finding the ground state of spin Hamiltonians with reinforcement learning

TL;DR

This paper introduces a reinforcement learning approach to optimize simulated annealing for finding ground states of spin Hamiltonians, outperforming traditional schedules and generalizing across Hamiltonian classes.

Contribution

The paper presents a novel RL-based method that learns adaptive temperature schedules for annealing, surpassing heuristic methods and demonstrating scalability and robustness.

Findings

RL-driven annealing outperforms standard schedules

The method generalizes to unseen Hamiltonians within a class

Achieves an order of magnitude improvement in scaling performance

Abstract

Reinforcement learning (RL) has become a proven method for optimizing a procedure for which success has been defined, but the specific actions needed to achieve it have not. We apply the so-called "black box" method of RL to what has been referred as the "black art" of simulated annealing (SA), demonstrating that an RL agent based on proximal policy optimization can, through experience alone, arrive at a temperature schedule that surpasses the performance of standard heuristic temperature schedules for two classes of Hamiltonians. When the system is initialized at a cool temperature, the RL agent learns to heat the system to "melt" it, and then slowly cool it in an effort to anneal to the ground state; if the system is initialized at a high temperature, the algorithm immediately cools the system. We investigate the performance of our RL-driven SA agent in generalizing to all Hamiltonians of a specific class; when trained on random Hamiltonians of nearest-neighbour spin glasses, the RL agent is able to control the SA process for other Hamiltonians, reaching the ground state with a higher probability than a simple linear annealing schedule. Furthermore, the scaling performance (with respect to system size) of the RL approach is far more favourable, achieving a performance improvement of one order of magnitude on L=14x14 systems. We demonstrate the robustness of the RL approach when the system operates in a "destructive observation" mode, an allusion to a quantum system where measurements destroy the state of the system. The success of the RL agent could have far-reaching impact, from classical optimization, to quantum annealing, to the simulation of physical systems.