RL-ABC: Reinforcement Learning for Accelerator Beamline Control

TL;DR

RL-ABC is an open-source framework that applies reinforcement learning to optimize particle accelerator beamlines, transforming complex control problems into Markov decision processes with minimal expert intervention.

Contribution

It introduces a general methodology for formulating beamline tuning as a Markov decision process and provides a flexible, integrated RL framework compatible with existing simulation tools.

Findings

RL-ABC successfully optimized a test beamline with 37 control parameters.

A Deep Deterministic Policy Gradient agent achieved 70.3% particle transmission.

Stage learning improved training efficiency by decomposing complex problems.

Abstract

Particle accelerator beamline optimization is a high-dimensional control problem traditionally requiring significant expert intervention. We present RLABC (Reinforcement Learning for Accelerator Beamline Control), an open-source Python framework that automatically transforms standard Elegant beamline configurations into reinforcement learning environments. RLABC integrates with the widely-used Elegant beam dynamics simulation code via SDDS-based interfaces, enabling researchers to apply modern RL algorithms to beamline optimization with minimal RL-specific development. The main contribution is a general methodology for formulating beamline tuning as a Markov decision process: RLABC automatically preprocesses lattice files to insert diagnostic watch points before each tunable element, constructs a 57-dimensional state representation from beam statistics, covariance information, and aperture constraints, and provides a configurable reward function for transmission optimization. The framework supports multiple RL algorithms through Stable-Baselines3 compatibility and implements stage learning strategies for improved training efficiency. Validation on a test beamline derived from the VEPP-5 injection complex (37 control parameters across 11 quadrupoles and 4 dipoles) demonstrates that the framework successfully enables RL-based optimization, with a Deep Deterministic Policy Gradient agent achieving 70.3\% particle transmission -- performance matching established methods such as differential evolution. The framework's stage learning capability allows decomposition of complex optimization problems into manageable subproblems, improving training efficiency. The complete framework, including configuration files and example notebooks, is available as open-source software to facilitate adoption and further research.