Robust Deep Reinforcement Learning for Volt-VAR Optimization in Active Distribution System under Uncertainty

TL;DR

This paper introduces a robust deep reinforcement learning framework for Volt-VAR optimization in active distribution networks, effectively managing uncertainties and ensuring safe operation with improved sample efficiency.

Contribution

It develops a robust DDPG algorithm that handles hybrid control actions and uncertainties using conformal prediction and adversarial modeling, enhancing safety and efficiency.

Findings

Demonstrates improved safety over benchmark algorithms.

Shows effective management of uncertainties in IEEE test cases.

Achieves higher sample efficiency in VVO tasks.

Abstract

The deep reinforcement learning (DRL) based Volt-VAR optimization (VVO) methods have been widely studied for active distribution networks (ADNs). However, most of them lack safety guarantees in terms of power injection uncertainties due to the increase in distributed energy resources (DERs) and load demand, such as electric vehicles. This article proposes a robust deep reinforcement learning (RDRL) framework for VVO via a robust deep deterministic policy gradient (DDPG) algorithm. This algorithm can effectively manage hybrid action spaces, considering control devices like capacitors, voltage regulators, and smart inverters. Additionally, it is designed to handle uncertainties by quantifying uncertainty sets with conformal prediction and modeling uncertainties as adversarial attacks to guarantee safe exploration across action spaces. Numerical results on three IEEE test cases demonstrate the sample efficiency and safety of the proposed robust DDPG against uncertainties compared to the benchmark algorithms.