High-Fidelity Data-Driven Dynamics Model for Reinforcement Learning-based Control in HL-3 Tokamak

TL;DR

This paper develops a high-fidelity, data-driven plasma dynamics model that enables rapid reinforcement learning-based control in tokamaks, demonstrating accurate, robust, and fast trajectory control suitable for future fusion devices like ITER.

Contribution

It introduces a novel data-driven simulator that mitigates autoregressive errors, enabling efficient RL training for plasma control in tokamaks.

Findings

Achieved 400-ms, 1 kHz plasma trajectory control with accurate waveform tracking.

Demonstrated zero-shot adaptation to changed plasma triangularity targets.

Enabled fast RL training suitable for real-time fusion device operation.

Abstract

The success of reinforcement learning (RL)-based control in tokamaks, an emerging technique for controlled nuclear fusion with improved flexibility, typically requires substantial interaction with a simulator capable of accurately evolving the high-dimensional plasma state. Compared to first-principle-based simulators, whose intense computations lead to sluggish RL training, we devise an effective method to acquire a fully data-driven simulator, by mitigating the arising compounding error issue due to the underlying autoregressive nature. With high accuracy and appealing extrapolation capability, this high-fidelity dynamics model subsequently enables the rapid training of a qualified RL agent to directly generate engineering-reasonable actuator commands, aiming at the desired long-term targets of plasma configuration. Together with a surrogate model for Equilibrium Fitting code based on neural network, named EFITNN, the RL agent successfully maintains a 400-ms, 1 kHz trajectory control with accurate waveform tracking of plasma current and last closed flux surface on the HL-3 tokamak. Furthermore, it also demonstrates the feasibility of zero-shot adaptation to changed triangularity targets, confirming the robustness of the developed data-driven dynamics model. Our work underscores the advantage of fully data-driven dynamics models in yielding RL-based trajectory control policies at a sufficiently fast pace, an anticipated engineering requirement in daily discharge practices for the upcoming ITER device.