Explaining Unreliable Perception in Automated Driving: A Fuzzy-based Monitoring Approach

TL;DR

This paper presents a fuzzy-based runtime monitor for ML perception in autonomous driving that offers human-interpretable explanations of perception reliability and improves safety assurance compared to existing methods.

Contribution

It introduces a novel fuzzy-based monitor that explains perception errors and enhances safety monitoring in autonomous driving systems.

Findings

The monitor provides interpretable insights into perception reliability under various conditions.

It improves safety by reducing hazardous situations in autonomous driving scenarios.

The approach maintains system availability while increasing safety compared to state-of-the-art monitors.

Abstract

Autonomous systems that rely on Machine Learning (ML) utilize online fault tolerance mechanisms, such as runtime monitors, to detect ML prediction errors and maintain safety during operation. However, the lack of human-interpretable explanations for these errors can hinder the creation of strong assurances about the system's safety and reliability. This paper introduces a novel fuzzy-based monitor tailored for ML perception components. It provides human-interpretable explanations about how different operating conditions affect the reliability of perception components and also functions as a runtime safety monitor. We evaluated our proposed monitor using naturalistic driving datasets as part of an automated driving case study. The interpretability of the monitor was evaluated and we identified a set of operating conditions in which the perception component performs reliably. Additionally, we created an assurance case that links unit-level evidence of \textit{correct} ML operation to system-level \textit{safety}. The benchmarking demonstrated that our monitor achieved a better increase in safety (i.e., absence of hazardous situations) while maintaining availability (i.e., ability to perform the mission) compared to state-of-the-art runtime ML monitors in the evaluated dataset.