Safe Wasserstein Constrained Deep Q-Learning

TL;DR

This paper introduces a distributionally robust Q-Learning algorithm that uses Wasserstein ambiguity sets to ensure probabilistic safety guarantees during online learning, demonstrated through a lithium-ion battery charging case study.

Contribution

The paper proposes DrQ, a novel safe reinforcement learning method that incorporates Wasserstein distributionally robust optimization to provide out-of-sample safety guarantees.

Findings

DrQ improves safety over traditional methods in battery charging.

Wasserstein DRO effectively bounds worst-case modeling errors.

The approach enhances safe exploration in constrained MDPs.

Abstract

This paper presents a distributionally robust Q-Learning algorithm (DrQ) which leverages Wasserstein ambiguity sets to provide idealistic probabilistic out-of-sample safety guarantees during online learning. First, we follow past work by separating the constraint functions from the principal objective to create a hierarchy of machines which estimate the feasible state-action space within the constrained Markov decision process (CMDP). DrQ works within this framework by augmenting constraint costs with tightening offset variables obtained through Wasserstein distributionally robust optimization (DRO). These offset variables correspond to worst-case distributions of modeling error characterized by the TD-errors of the constraint Q-functions. This procedure allows us to safely approach the nominal constraint boundaries. Using a case study of lithium-ion battery fast charging, we explore how idealistic safety guarantees translate to generally improved safety relative to conventional methods.