Offline Reinforcement Learning for Rotation Profile Control in Tokamaks

TL;DR

This paper explores offline reinforcement learning to control plasma rotation profiles in tokamaks, using historical data and probabilistic models to develop policies deployed successfully on the DIII-D device.

Contribution

It introduces an offline RL approach with probabilistic models for rotation control, demonstrating real-world deployment on a tokamak using only past data.

Findings

RL policies achieved promising control results on DIII-D

Probabilistic models enabled effective offline training

Insights into challenges of deploying RL in physical fusion devices

Abstract

Tokamaks remain leading candidates for achieving practical fusion energy, yet many important control problems inside these devices are still difficult or unsolved. One such challenge is controlling the plasma rotation profile, which strongly influences stability, confinement, and transport. While the average rotation can be controlled, controlling the full profile is challenging due to high dimensionality, response to multiple actuators and dependence on plasma condition. Learning-based control methods, such as reinforcement learning (RL), provide a potential solution to this challenging problem with ability to model complex interactions leading to effective multi-input multi-output control. However, learning such policies is challenging due to the lack of accurate simulators that can model the rotation profile dynamics. In this work, we investigate the use of offline RL and offline model-based RL algorithms for rotation profile control, training them solely on historical data from the DIII-D tokamak. Our final method uses probabilistic models of plasma dynamics to generate rollouts for RL training. We deploy this policy on the DIII-D Tokamak and observe promising real-world results. We conclude by highlighting key challenges and insights from training and deploying an RL policy on a complex physical device while using only limited past data.