Surpassing legacy approaches to PWR core reload optimization with single-objective Reinforcement learning

TL;DR

This paper demonstrates that Deep Reinforcement Learning, specifically Proximal Policy Optimization (PPO), outperforms traditional stochastic optimization methods in nuclear reactor core reload pattern optimization, offering a more effective approach for complex, multi-objective problems.

Contribution

The study introduces a DRL-based method using PPO for core reload optimization, showing its superiority over common stochastic algorithms in efficiency and solution quality.

Findings

PPO outperforms genetic algorithms, simulated annealing, and tabu search.

PPO effectively balances global and local search capabilities.

Statistical analysis confirms PPO's superiority in long-term optimization runs.

Abstract

Optimizing the fuel cycle cost through the optimization of nuclear reactor core loading patterns involves multiple objectives and constraints, leading to a vast number of candidate solutions that cannot be explicitly solved. To advance the state-of-the-art in core reload patterns, we have developed methods based on Deep Reinforcement Learning (DRL) for both single- and multi-objective optimization. Our previous research has laid the groundwork for these approaches and demonstrated their ability to discover high-quality patterns within a reasonable time frame. On the other hand, stochastic optimization (SO) approaches are commonly used in the literature, but there is no rigorous explanation that shows which approach is better in which scenario. In this paper, we demonstrate the advantage of our RL-based approach, specifically using Proximal Policy Optimization (PPO), against the most commonly used SO-based methods: Genetic Algorithm (GA), Parallel Simulated Annealing (PSA) with mixing of states, and Tabu Search (TS), as well as an ensemble-based method, Prioritized Replay Evolutionary and Swarm Algorithm (PESA). We found that the LP scenarios derived in this paper are amenable to a global search to identify promising research directions rapidly, but then need to transition into a local search to exploit these directions efficiently and prevent getting stuck in local optima. PPO adapts its search capability via a policy with learnable weights, allowing it to function as both a global and local search method. Subsequently, we compared all algorithms against PPO in long runs, which exacerbated the differences seen in the shorter cases. Overall, the work demonstrates the statistical superiority of PPO compared to the other considered algorithms.