On Training Effective Reinforcement Learning Agents for Real-time Power Grid Operation and Control

TL;DR

This paper introduces a novel AI-based reinforcement learning approach using the soft actor-critic algorithm for real-time power grid control, enabling fast, multi-objective regulation of voltages and transmission losses in large-scale grids.

Contribution

It presents a new multi-objective RL methodology with offline and online training for real-world power grid control, validated through deployment and extensive numerical studies.

Findings

SAC agents effectively regulate voltage profiles.

Agents reduce transmission losses in real-time.

Control actions are provided within subsecond timescales.

Abstract

Deriving fast and effectively coordinated control actions remains a grand challenge affecting the secure and economic operation of today's large-scale power grid. This paper presents a novel artificial intelligence (AI) based methodology to achieve multi-objective real-time power grid control for real-world implementation. State-of-the-art off-policy reinforcement learning (RL) algorithm, soft actor-critic (SAC) is adopted to train AI agents with multi-thread offline training and periodic online training for regulating voltages and transmission losses without violating thermal constraints of lines. A software prototype was developed and deployed in the control center of SGCC Jiangsu Electric Power Company that interacts with their Energy Management System (EMS) every 5 minutes. Massive numerical studies using actual power grid snapshots in the real-time environment verify the effectiveness of the proposed approach. Well-trained SAC agents can learn to provide effective and subsecond control actions in regulating voltage profiles and reducing transmission losses.