Improving Autonomous Separation Assurance through Distributed Reinforcement Learning with Attention Networks

TL;DR

This paper introduces a decentralized reinforcement learning framework using attention networks for autonomous aircraft separation in dense, uncertain environments, enhancing safety and efficiency in advanced air mobility corridors.

Contribution

It develops a novel extension to the soft actor-critic algorithm incorporating attention mechanisms and distributed architecture for real-time, safety-critical aircraft separation.

Findings

Ensures safe separation in high-density environments

Achieves high training throughput with distributed computing

Handles various sources of uncertainty effectively

Abstract

Advanced Air Mobility (AAM) introduces a new, efficient mode of transportation with the use of vehicle autonomy and electrified aircraft to provide increasingly autonomous transportation between previously underserved markets. Safe and efficient navigation of low altitude aircraft through highly dense environments requires the integration of a multitude of complex observations, such as surveillance, knowledge of vehicle dynamics, and weather. The processing and reasoning on these observations pose challenges due to the various sources of uncertainty in the information while ensuring cooperation with a variable number of aircraft in the airspace. These challenges coupled with the requirement to make safety-critical decisions in real-time rule out the use of conventional separation assurance techniques. We present a decentralized reinforcement learning framework to provide autonomous self-separation capabilities within AAM corridors with the use of speed and vertical maneuvers. The problem is formulated as a Markov Decision Process and solved by developing a novel extension to the sample-efficient, off-policy soft actor-critic (SAC) algorithm. We introduce the use of attention networks for variable-length observation processing and a distributed computing architecture to achieve high training sample throughput as compared to existing approaches. A comprehensive numerical study shows that the proposed framework can ensure safe and efficient separation of aircraft in high density, dynamic environments with various sources of uncertainty.