Safety Monitoring of Machine Learning Perception Functions: a Survey

TL;DR

This survey reviews safety monitoring techniques for machine learning perception functions in safety-critical systems, emphasizing design considerations, challenges, and future research directions to ensure system dependability.

Contribution

It provides a comprehensive structured overview of existing safety monitoring methods for ML perception in safety-critical applications, highlighting key factors and challenges.

Findings

Identifies key factors in safety monitor design.

Highlights ongoing challenges in safety monitoring.

Suggests future research directions.

Abstract

Machine Learning (ML) models, such as deep neural networks, are widely applied in autonomous systems to perform complex perception tasks. New dependability challenges arise when ML predictions are used in safety-critical applications, like autonomous cars and surgical robots. Thus, the use of fault tolerance mechanisms, such as safety monitors, is essential to ensure the safe behavior of the system despite the occurrence of faults. This paper presents an extensive literature review on safety monitoring of perception functions using ML in a safety-critical context. In this review, we structure the existing literature to highlight key factors to consider when designing such monitors: threat identification, requirements elicitation, detection of failure, reaction, and evaluation. We also highlight the ongoing challenges associated with safety monitoring and suggest directions for future research.