Towards practical reinforcement learning for tokamak magnetic control

TL;DR

This paper advances reinforcement learning for tokamak magnetic control by improving accuracy, reducing steady-state error, and decreasing training time, validated through simulation and experiments on the TCV tokamak.

Contribution

It introduces algorithmic improvements to RL agents for plasma control, enhancing accuracy and training efficiency over previous methods.

Findings

Up to 65% improvement in shape accuracy

Significant reduction in plasma current bias

Training time reduced by a factor of 3 or more

Abstract

Reinforcement learning (RL) has shown promising results for real-time control systems, including the domain of plasma magnetic control. However, there are still significant drawbacks compared to traditional feedback control approaches for magnetic confinement. In this work, we address key drawbacks of the RL method; achieving higher control accuracy for desired plasma properties, reducing the steady-state error, and decreasing the required time to learn new tasks. We build on top of \cite{degrave2022magnetic}, and present algorithmic improvements to the agent architecture and training procedure. We present simulation results that show up to 65\% improvement in shape accuracy, achieve substantial reduction in the long-term bias of the plasma current, and additionally reduce the training time required to learn new tasks by a factor of 3 or more. We present new experiments using the upgraded RL-based controllers on the TCV tokamak, which validate the simulation results achieved, and point the way towards routinely achieving accurate discharges using the RL approach.