Adaptive Stress Testing of Trajectory Predictions in Flight Management Systems

TL;DR

This paper applies adaptive stress testing with a modified Monte Carlo tree search to identify failure scenarios in flight trajectory prediction systems, improving detection of likely failures over traditional methods.

Contribution

It extends adaptive stress testing to sequential decision-making problems and demonstrates its effectiveness in finding more and higher likelihood failures in flight management systems.

Findings

Adaptive stress testing finds more failures than baseline methods.

The approach identifies failures with higher estimated likelihood.

It effectively uncovers potential issues before deployment.

Abstract

To find failure events and their likelihoods in flight-critical systems, we investigate the use of an advanced black-box stress testing approach called adaptive stress testing. We analyze a trajectory predictor from a developmental commercial flight management system which takes as input a collection of lateral waypoints and en-route environmental conditions. Our aim is to search for failure events relating to inconsistencies in the predicted lateral trajectories. The intention of this work is to find likely failures and report them back to the developers so they can address and potentially resolve shortcomings of the system before deployment. To improve search performance, this work extends the adaptive stress testing formulation to be applied more generally to sequential decision-making problems with episodic reward by collecting the state transitions during the search and evaluating at the end of the simulated rollout. We use a modified Monte Carlo tree search algorithm with progressive widening as our adversarial reinforcement learner. The performance is compared to direct Monte Carlo simulations and to the cross-entropy method as an alternative importance sampling baseline. The goal is to find potential problems otherwise not found by traditional requirements-based testing. Results indicate that our adaptive stress testing approach finds more failures and finds failures with higher likelihood relative to the baseline approaches.