Safety Enhancement for Deep Reinforcement Learning in Autonomous Separation Assurance

TL;DR

This paper introduces a safety module for deep reinforcement learning in autonomous air traffic separation, addressing model and state uncertainties to enhance safety in complex environments.

Contribution

It proposes a novel safety framework with two sub-modules that improve DRL safety performance in autonomous separation tasks under uncertainty.

Findings

The safety modules significantly reduce risk in separation assurance.

The Monte-Carlo dropout improves model uncertainty estimation.

State data augmentation enhances robustness against disturbances.

Abstract

The separation assurance task will be extremely challenging for air traffic controllers in a complex and high density airspace environment. Deep reinforcement learning (DRL) was used to develop an autonomous separation assurance framework in our previous work where the learned model advised speed maneuvers. In order to improve the safety of this model in unseen environments with uncertainties, in this work we propose a safety module for DRL in autonomous separation assurance applications. The proposed module directly addresses both model uncertainty and state uncertainty to improve safety. Our safety module consists of two sub-modules: (1) the state safety sub-module is based on the execution-time data augmentation method to introduce state disturbances in the model input state; (2) the model safety sub-module is a Monte-Carlo dropout extension that learns the posterior distribution of the DRL model policy. We demonstrate the effectiveness of the two sub-modules in an open-source air traffic simulator with challenging environment settings. Through extensive numerical experiments, our results show that the proposed sub-safety modules help the DRL agent significantly improve its safety performance in an autonomous separation assurance task.