A Review of Safe Reinforcement Learning Methods for Modern Power Systems

TL;DR

This paper reviews safe reinforcement learning techniques tailored for modern power systems, emphasizing safety guarantees, practical applications, and addressing challenges like scalability and robustness for real-world deployment.

Contribution

It provides a comprehensive overview of safe RL methods, their applications in power systems, and discusses challenges and future directions for reliable implementation.

Findings

Safe RL methods incorporate safety layers and Lyapunov functions.

Techniques like Lagrangian relaxation enhance safety guarantees.

Real-world case studies demonstrate practical deployment.

Abstract

Given the availability of more comprehensive measurement data in modern power systems, reinforcement learning (RL) has gained significant interest in operation and control. Conventional RL relies on trial-and-error interactions with the environment and reward feedback, which often leads to exploring unsafe operating regions and executing unsafe actions, especially when deployed in real-world power systems. To address these challenges, safe RL has been proposed to optimize operational objectives while ensuring safety constraints are met, keeping actions and states within safe regions throughout both training and deployment. Rather than relying solely on manually designed penalty terms for unsafe actions, as is common in conventional RL, safe RL methods reviewed here primarily leverage advanced and proactive mechanisms. These include techniques such as Lagrangian relaxation, safety layers, and theoretical guarantees like Lyapunov functions to rigorously enforce safety boundaries. This paper provides a comprehensive review of safe RL methods and their applications across various power system operations and control domains, including security control, real-time operation, operational planning, and emerging areas. It summarizes existing safe RL techniques, evaluates their performance, analyzes suitable deployment scenarios, and examines algorithm benchmarks and application environments. The paper also highlights real-world implementation cases and identifies critical challenges such as scalability in large-scale systems and robustness under uncertainty, providing potential solutions and outlining future directions to advance the reliable integration and deployment of safe RL in modern power systems.