Adaptive Model-Based Reinforcement Learning for Orbit Feedback Control in NSLS-II Storage Ring

TL;DR

This paper introduces an adaptive model-based reinforcement learning framework for orbit feedback control in NSLS-II, effectively addressing noise, drifting, and non-linear dynamics to improve beam stability and alignment accuracy.

Contribution

It presents a novel adaptive reinforcement learning approach that combines trajectory and model optimization for real-time accelerator control, outperforming traditional methods.

Findings

Stabilizes beam position with RMS error ~1μm.

Effectively tracks internal status drifting.

Reduces alignment error compared to traditional algorithms.

Abstract

The National Synchrotron Light Source II (NSLS-II) uses highly stable electron beam to produce high-quality X-ray beams with high brightness and low-emittance synchrotron radiation. The traditional algorithm to stabilize the beam applies singular value decomposition (SVD) on the orbit response matrix to remove noise and extract actions. Supervised learning has been studied on NSLS-II storage ring stabilization and other accelerator facilities recently. Several problems, for example, machine status drifting, environment noise, and non-linear accelerator dynamics, remain unresolved in the SVD-based and supervised learning algorithms. To address these problems, we propose an adaptive training framework based on model-based reinforcement learning. This framework consists of two types of optimizations: trajectory optimization attempts to minimize the expected total reward in a differentiable environment, and online model optimization learns non-linear machine dynamics through the agent-environment interaction. Through online training, this framework tracks the internal status drifting in the electron beam ring. Simulation and real in-facility experiments on NSLS-II reveal that our method stabilizes the beam position and minimizes the alignment error, defined as the root mean square (RMS) error between adjusted beam positions and the reference position, down to ~1$\mu$m.