Towards Safe Machine Learning for CPS: Infer Uncertainty from Training Data

TL;DR

This paper introduces a Feature Space Partitioning Tree (FSPT) to identify the training space in machine learning models for cyber-physical systems, enhancing safety by detecting when models are extrapolating beyond learned data.

Contribution

The paper proposes a novel FSPT method to efficiently partition feature space and determine training data coverage, improving safety in ML for CPS.

Findings

FSPT effectively identifies regions with sufficient training data.

Model performance correlates strongly with FSPT scores.

FSPT helps prevent unsafe extrapolations in ML models.

Abstract

Machine learning (ML) techniques are increasingly applied to decision-making and control problems in Cyber-Physical Systems among which many are safety-critical, e.g., chemical plants, robotics, autonomous vehicles. Despite the significant benefits brought by ML techniques, they also raise additional safety issues because 1) most expressive and powerful ML models are not transparent and behave as a black box and 2) the training data which plays a crucial role in ML safety is usually incomplete. An important technique to achieve safety for ML models is "Safe Fail", i.e., a model selects a reject option and applies the backup solution, a traditional controller or a human operator for example, when it has low confidence in a prediction. Data-driven models produced by ML algorithms learn from training data, and hence they are only as good as the examples they have learnt. As pointed in [17], ML models work well in the "training space" (i.e., feature space with sufficient training data), but they could not extrapolate beyond the training space. As observed in many previous studies, a feature space that lacks training data generally has a much higher error rate than the one that contains sufficient training samples [31]. Therefore, it is essential to identify the training space and avoid extrapolating beyond the training space. In this paper, we propose an efficient Feature Space Partitioning Tree (FSPT) to address this problem. Using experiments, we also show that, a strong relationship exists between model performance and FSPT score.