Hierarchical Reinforcement Learning with Runtime Safety Shielding for Power Grid Operation

TL;DR

This paper introduces a hierarchical reinforcement learning framework with a runtime safety shield for power grid control, enhancing safety, robustness, and generalization without retraining.

Contribution

It proposes a novel hierarchical control architecture that decouples decision-making from safety enforcement, enabling safe and robust power grid operation in unseen scenarios.

Findings

Hierarchical approach outperforms flat RL under stress tests.

Safety shield ensures real-time safety invariants.

Zero-shot deployment achieves robust performance without retraining.

Abstract

Reinforcement learning has shown promise for automating power-grid operation tasks such as topology control and congestion management. However, its deployment in real-world power systems remains limited by strict safety requirements, brittleness under rare disturbances, and poor generalization to unseen grid topologies. In safety-critical infrastructure, catastrophic failures cannot be tolerated, and learning-based controllers must operate within hard physical constraints. This paper proposes a safety-constrained hierarchical control framework for power-grid operation that explicitly decouples long-horizon decision-making from real-time feasibility enforcement. A high-level reinforcement learning policy proposes abstract control actions, while a deterministic runtime safety shield filters unsafe actions using fast forward simulation. Safety is enforced as a runtime invariant, independent of policy quality or training distribution. The proposed framework is evaluated on the Grid2Op benchmark suite under nominal conditions, forced line-outage stress tests, and zero-shot deployment on the ICAPS 2021 large-scale transmission grid without retraining. Results show that flat reinforcement learning policies are brittle under stress, while safety-only methods are overly conservative. In contrast, the proposed hierarchical and safety-aware approach achieves longer episode survival, lower peak line loading, and robust zero-shot generalization to unseen grids. These results indicate that safety and generalization in power-grid control are best achieved through architectural design rather than increasingly complex reward engineering, providing a practical path toward deployable learning-based controllers for real-world energy systems.