Relating System Safety and Machine Learnt Model Performance

TL;DR

This paper explores how to connect safety objectives with machine learning model performance metrics in aeronautical systems, proposing a method to derive safety-related requirements for neural networks.

Contribution

It introduces a method to relate safety goals to model performance metrics, providing a framework for safety-driven model development in aeronautics.

Findings

Proposes a simple abstraction of MLC behavior for safety assessment.

Provides a method to derive safety-related performance requirements.

Clarifies assumptions and constraints for applying the method.

Abstract

The prediction quality of machine learnt models and the functionality they ultimately enable (e.g., object detection), is typically evaluated using a variety of quantitative metrics that are specified in the associated model performance requirements. When integrating such models into aeronautical applications, a top-down safety assessment process must influence both the model performance metrics selected, and their acceptable range of values. Often, however, the relationship of system safety objectives to model performance requirements and the associated metrics is unclear. Using an example of an aircraft emergency braking system containing a machine learnt component (MLC) responsible for object detection and alerting, this paper first describes a simple abstraction of the required MLC behavior. Then, based on that abstraction, an initial method is given to derive the minimum safety-related performance requirements, the associated metrics, and their targets for the both MLC and its underlying deep neural network, such that they meet the quantitative safety objectives obtained from the safety assessment process. We give rationale as to why the proposed method should be considered valid, also clarifying the assumptions made, the constraints on applicability, and the implications for verification.