Conformal Safety Monitoring for Flight Testing: A Case Study in Data-Driven Safety Learning

TL;DR

This paper presents a data-driven safety monitoring method for flight testing that predicts and classifies short-term safety risks using conformal prediction, improving preemptive safety decision-making.

Contribution

It introduces a novel combination of stochastic simulation, predictive modeling, and conformal calibration for real-time safety monitoring in uncertain flight scenarios.

Findings

Reliable identification of unsafe flight scenarios

Matches theoretical safety guarantees

Outperforms baseline risk classification methods

Abstract

We develop a data-driven approach for runtime safety monitoring in flight testing, where pilots perform maneuvers on aircraft with uncertain parameters. Because safety violations can arise unexpectedly as a result of these uncertainties, pilots need clear, preemptive criteria to abort the maneuver in advance of safety violation. To solve this problem, we use offline stochastic trajectory simulation to learn a calibrated statistical model of the short-term safety risk facing pilots. We use flight testing as a motivating example for data-driven learning/monitoring of safety due to its inherent safety risk, uncertainty, and human-interaction. However, our approach consists of three broadly-applicable components: a model to predict future state from recent observations, a nearest neighbor model to classify the safety of the predicted state, and classifier calibration via conformal prediction. We evaluate our method on a flight dynamics model with uncertain parameters, demonstrating its ability to reliably identify unsafe scenarios, match theoretical guarantees, and outperform baseline approaches in preemptive classification of risk.