A Critical Review of Safe Reinforcement Learning Techniques in Smart Grid Applications

TL;DR

This paper reviews recent safe reinforcement learning methods applied to smart power systems, emphasizing safety challenges, algorithmic approaches, and future research opportunities in managing uncertainties from distributed energy resources.

Contribution

It provides a comprehensive review of safe RL techniques in power systems, highlighting their applications, limitations, and potential for enhancing safety in uncertain environments.

Findings

Safe RL algorithms are increasingly applied in power system control.

Current methods face bottlenecks in safety guarantees and scalability.

Future research should address safety, efficiency, and real-time implementation challenges.

Abstract

The high penetration of distributed energy resources (DERs) in modern smart power systems introduces unforeseen uncertainties for the electricity sector, leading to increased complexity and difficulty in the operation and control of power systems. As a cutting-edge machine learning technology, deep reinforcement learning (DRL) has been widely implemented in recent years to handle the uncertainty in power systems. However, in critical infrastructures such as power systems, safety issues always receive top priority, while DRL may not always meet the safety requirements of power system operators. The concept of safe reinforcement learning (safe RL) is emerging as a potential solution to overcome the shortcomings of conventional DRL in the operation and control of power systems. This study provides a rigorous review of the latest research efforts focused on safe RL to derive power system control policies while accounting for the unique safety requirements of power grids. Furthermore, this study highlights various safe RL algorithms applied in diverse applications within the power system sector, from single grid-connected power converters, residential smart homes, and buildings to large power distribution networks. For all methods outlined, a discussion on their bottlenecks, research challenges, and potential opportunities in the operation and control of power system applications is also presented. This review aims to support research in the area of safe RL algorithms, embracing smart power system operation with safety constraints amid high uncertainty from DERs.