Incorporating Failure of Machine Learning in Dynamic Probabilistic Safety Assurance

TL;DR

This paper presents a probabilistic safety assurance framework that integrates SafeML with Bayesian Networks to dynamically evaluate and adapt safety in ML-based systems under distributional shifts, demonstrated on autonomous vehicle platooning.

Contribution

It introduces a novel safety assessment approach combining SafeML and Bayesian Networks to model and manage ML failures in safety-critical systems.

Findings

Effective detection of distributional shifts using SafeML

Bayesian Networks enable dynamic safety evaluation under uncertainty

Application to automotive platooning shows practical benefits

Abstract

Machine Learning (ML) models are increasingly integrated into safety-critical systems, such as autonomous vehicle platooning, to enable real-time decision-making. However, their inherent imperfection introduces a new class of failure: reasoning failures often triggered by distributional shifts between operational and training data. Traditional safety assessment methods, which rely on design artefacts or code, are ill-suited for ML components that learn behaviour from data. SafeML was recently proposed to dynamically detect such shifts and assign confidence levels to the reasoning of ML-based components. Building on this, we introduce a probabilistic safety assurance framework that integrates SafeML with Bayesian Networks (BNs) to model ML failures as part of a broader causal safety analysis. This allows for dynamic safety evaluation and system adaptation under uncertainty. We demonstrate the approach on an simulated automotive platooning system with traffic sign recognition. The findings highlight the potential broader benefits of explicitly modelling ML failures in safety assessment.