Learning Run-time Safety Monitors for Machine Learning Components

TL;DR

This paper presents a method to create runtime safety monitors for machine learning components in autonomous systems, using degraded datasets and machine learning to predict safety risks without ground truth at runtime.

Contribution

It introduces a novel process for developing safety monitors that operate in parallel with ML components to assess safety risks during deployment.

Findings

Safety monitors successfully predict risks in experiments

Method works with publicly available datasets

Potential to enhance autonomous system safety

Abstract

For machine learning components used as part of autonomous systems (AS) in carrying out critical tasks it is crucial that assurance of the models can be maintained in the face of post-deployment changes (such as changes in the operating environment of the system). A critical part of this is to be able to monitor when the performance of the model at runtime (as a result of changes) poses a safety risk to the system. This is a particularly difficult challenge when ground truth is unavailable at runtime. In this paper we introduce a process for creating safety monitors for ML components through the use of degraded datasets and machine learning. The safety monitor that is created is deployed to the AS in parallel to the ML component to provide a prediction of the safety risk associated with the model output. We demonstrate the viability of our approach through some initial experiments using publicly available speed sign datasets.