Is the Rush to Machine Learning Jeopardizing Safety? Results of a Survey

TL;DR

This paper surveys the rapid adoption of machine learning in safety-critical systems, highlighting potential safety concerns and the lack of thorough safety evaluations compared to research efforts.

Contribution

It provides an analysis of the disparity between ML application in safety-critical systems and the safety assessment research dedicated to these systems.

Findings

Limited safety evaluation research on ML in safety-critical systems

Rapid deployment of ML components without comprehensive safety validation

Potential safety risks due to insufficient safety standards adaptation

Abstract

Machine learning (ML) is finding its way into safety-critical systems (SCS). Current safety standards and practice were not designed to cope with ML techniques, and it is difficult to be confident that SCSs that contain ML components are safe. Our hypothesis was that there has been a rush to deploy ML techniques at the expense of a thorough examination as to whether the use of ML techniques introduces safety problems that we are not yet adequately able to detect and mitigate against. We thus conducted a targeted literature survey to determine the research effort that has been expended in applying ML to SCS compared with that spent on evaluating the safety of SCSs that deploy ML components. This paper presents the (surprising) results of the survey.