Neural Network Compression of ACAS Xu Early Prototype is Unsafe: Closed-Loop Verification through Quantized State Backreachability

TL;DR

This paper presents a method for verifying the safety of a neural network-based collision avoidance system in unmanned aircraft by using closed-loop analysis with state quantization and backreachability, addressing limitations of open-loop verification.

Contribution

It introduces a novel closed-loop verification technique for neural network controllers in safety-critical systems, incorporating state quantization and backreachability analysis.

Findings

The method can verify safety under idealized assumptions.

Counterexamples are generated when safety cannot be proven.

Refinement of quantization parameters improves verification accuracy.

Abstract

ACAS Xu is an air-to-air collision avoidance system designed for unmanned aircraft that issues horizontal turn advisories to avoid an intruder aircraft. Due the use of a large lookup table in the design, a neural network compression of the policy was proposed. Analysis of this system has spurred a significant body of research in the formal methods community on neural network verification. While many powerful methods have been developed, most work focuses on open-loop properties of the networks, rather than the main point of the system -- collision avoidance -- which requires closed-loop analysis. In this work, we develop a technique to verify a closed-loop approximation of the system using state quantization and backreachability. We use favorable assumptions for the analysis -- perfect sensor information, instant following of advisories, ideal aircraft maneuvers and an intruder that only flies straight. When the method fails to prove the system is safe, we refine the quantization parameters until generating counterexamples where the original (non-quantized) system also has collisions.