On the Safety of Machine Learning: Cyber-Physical Systems, Decision Sciences, and Data Products

TL;DR

This paper formalizes the concept of safety in machine learning systems, analyzing risks and uncertainties across applications, and discusses strategies and techniques to enhance safety in cyber-physical systems and data products.

Contribution

It introduces a formal definition of machine learning safety based on risk and uncertainty, and maps traditional safety strategies to ML contexts with practical techniques.

Findings

Empirical risk minimization alone is insufficient for safety.

Interpretability and causality are crucial for safe ML models.

Human involvement and user experience design enhance safety.

Abstract

Machine learning algorithms increasingly influence our decisions and interact with us in all parts of our daily lives. Therefore, just as we consider the safety of power plants, highways, and a variety of other engineered socio-technical systems, we must also take into account the safety of systems involving machine learning. Heretofore, the definition of safety has not been formalized in a machine learning context. In this paper, we do so by defining machine learning safety in terms of risk, epistemic uncertainty, and the harm incurred by unwanted outcomes. We then use this definition to examine safety in all sorts of applications in cyber-physical systems, decision sciences, and data products. We find that the foundational principle of modern statistical machine learning, empirical risk minimization, is not always a sufficient objective. Finally, we discuss how four different categories of strategies for achieving safety in engineering, including inherently safe design, safety reserves, safe fail, and procedural safeguards can be mapped to a machine learning context. We then discuss example techniques that can be adopted in each category, such as considering interpretability and causality of predictive models, objective functions beyond expected prediction accuracy, human involvement for labeling difficult or rare examples, and user experience design of software and open data.