Causal Bayesian Networks for Data-driven Safety Analysis of Complex Systems

TL;DR

This paper introduces a method using causal Bayesian networks for safety analysis of complex systems, enabling better understanding of fault propagation and interaction with the environment, especially in safety-critical applications like automated driving.

Contribution

It proposes a novel approach combining causal Bayesian networks with safety analysis, comparing it to fault tree analysis, and demonstrating its application in automated driving systems.

Findings

Causal Bayesian networks provide a more comprehensive safety analysis than traditional fault trees.

The approach effectively models causal influences in complex, open-environment systems.

Evaluation on an automated driving perception system shows promising results.

Abstract

Ensuring safe operation of safety-critical complex systems interacting with their environment poses significant challenges, particularly when the system's world model relies on machine learning algorithms to process the perception input. A comprehensive safety argumentation requires knowledge of how faults or functional insufficiencies propagate through the system and interact with external factors, to manage their safety impact. While statistical analysis approaches can support the safety assessment, associative reasoning alone is neither sufficient for the safety argumentation nor for the identification and investigation of safety measures. A causal understanding of the system and its interaction with the environment is crucial for safeguarding safety-critical complex systems. It allows to transfer and generalize knowledge, such as insights gained from testing, and facilitates the identification of potential improvements. This work explores using causal Bayesian networks to model the system's causalities for safety analysis, and proposes measures to assess causal influences based on Pearl's framework of causal inference. We compare the approach of causal Bayesian networks to the well-established fault tree analysis, outlining advantages and limitations. In particular, we examine importance metrics typically employed in fault tree analysis as foundation to discuss suitable causal metrics. An evaluation is performed on the example of a perception system for automated driving. Overall, this work presents an approach for causal reasoning in safety analysis that enables the integration of data-driven and expert-based knowledge to account for uncertainties arising from complex systems operating in open environments.