Multistep Criticality Search and Power Shaping in Microreactors with Reinforcement Learning

TL;DR

This paper demonstrates the effectiveness of reinforcement learning algorithms, specifically PPO, for autonomous control of microreactors, achieving near-optimal power distribution and criticality maintenance in high-fidelity simulations.

Contribution

It introduces the application of advanced deep reinforcement learning techniques, PPO and A2C, for microreactor control, utilizing a surrogate model trained on detailed simulation data.

Findings

PPO achieved a power tilt ratio of approximately 1.002.

PPO maintained criticality within 10 pcm.

A2C was less effective than PPO in this control task.

Abstract

Reducing operation and maintenance costs is a key objective for advanced reactors in general and microreactors in particular. To achieve this reduction, developing robust autonomous control algorithms is essential to ensure safe and autonomous reactor operation. Recently, artificial intelligence and machine learning algorithms, specifically reinforcement learning (RL) algorithms, have seen rapid increased application to control problems, such as plasma control in fusion tokamaks and building energy management. In this work, we introduce the use of RL for intelligent control in nuclear microreactors. The RL agent is trained using proximal policy optimization (PPO) and advantage actor-critic (A2C), cutting-edge deep RL techniques, based on a high-fidelity simulation of a microreactor design inspired by the Westinghouse eVinci\textsuperscript{TM} design. We utilized a Serpent model to generate data on drum positions, core criticality, and core power distribution for training a feedforward neural network surrogate model. This surrogate model was then used to guide a PPO and A2C control policies in determining the optimal drum position across various reactor burnup states, ensuring critical core conditions and symmetrical power distribution across all six core portions. The results demonstrate the excellent performance of PPO in identifying optimal drum positions, achieving a hextant power tilt ratio of approximately 1.002 (within the limit of $<$ 1.02) and maintaining criticality within a 10 pcm range. A2C did not provide as competitive of a performance as PPO in terms of performance metrics for all burnup steps considered in the cycle. Additionally, the results highlight the capability of well-trained RL control policies to quickly identify control actions, suggesting a promising approach for enabling real-time autonomous control through digital twins.