Chance-Constrained Control with Lexicographic Deep Reinforcement Learning

TL;DR

This paper introduces a novel lexicographic Deep Reinforcement Learning method for chance-constrained Markov Decision Processes that simplifies training by separating constraint handling, avoiding hyper-parameter tuning, and ensuring probabilistic safety.

Contribution

It proposes a new approach that trains separate neural networks for constraints and system control, eliminating the need for weight tuning during training.

Findings

The method effectively manages chance constraints without hyper-parameter tuning.

The lexicographic DQN outperforms standard approaches in simulations.

The approach ensures probabilistic safety with minimal training complexity.

Abstract

This paper proposes a lexicographic Deep Reinforcement Learning (DeepRL)-based approach to chance-constrained Markov Decision Processes, in which the controller seeks to ensure that the probability of satisfying the constraint is above a given threshold. Standard DeepRL approaches require i) the constraints to be included as additional weighted terms in the cost function, in a multi-objective fashion, and ii) the tuning of the introduced weights during the training phase of the Deep Neural Network (DNN) according to the probability thresholds. The proposed approach, instead, requires to separately train one constraint-free DNN and one DNN associated to each constraint and then, at each time-step, to select which DNN to use depending on the system observed state. The presented solution does not require any hyper-parameter tuning besides the standard DNN ones, even if the probability thresholds changes. A lexicographic version of the well-known DeepRL algorithm DQN is also proposed and validated via simulations.