Evaluation of Runtime Monitoring for UAV Emergency Landing

TL;DR

This paper presents a new runtime monitoring approach for UAV emergency landing systems, demonstrating its safety benefits through a novel evaluation methodology and comparison with traditional mitigation strategies.

Contribution

It introduces a new emergency landing pipeline with runtime monitoring of learning components and a novel evaluation methodology for assessing safety in real-world UAV scenarios.

Findings

MLRM approaches improve safety over default parachute deployment

The proposed EL pipeline effectively monitors learning components during flight

Evaluation shows significant safety benefits of MLRM mechanisms

Abstract

To certify UAV operations in populated areas, risk mitigation strategies -- such as Emergency Landing (EL) -- must be in place to account for potential failures. EL aims at reducing ground risk by finding safe landing areas using on-board sensors. The first contribution of this paper is to present a new EL approach, in line with safety requirements introduced in recent research. In particular, the proposed EL pipeline includes mechanisms to monitor learning based components during execution. This way, another contribution is to study the behavior of Machine Learning Runtime Monitoring (MLRM) approaches within the context of a real-world critical system. A new evaluation methodology is introduced, and applied to assess the practical safety benefits of three MLRM mechanisms. The proposed approach is compared to a default mitigation strategy (open a parachute when a failure is detected), and appears to be much safer.