System Safety Monitoring of Learned Components Using Temporal Metric Forecasting

TL;DR

This paper introduces a safety monitoring approach for learned components in autonomous systems using probabilistic time series forecasting, demonstrating its effectiveness through case studies in aviation and driving.

Contribution

It proposes a novel safety monitoring method based on deep learning probabilistic forecasting, addressing challenges of limited internal access and real-time requirements.

Findings

TFT achieved highest accuracy in safety violation prediction

Probabilistic forecasting effectively predicts safety metrics

Method balances prediction accuracy with latency and resource use

Abstract

In learning-enabled autonomous systems, safety monitoring of learned components is crucial to ensure their outputs do not lead to system safety violations, given the operational context of the system. However, developing a safety monitor for practical deployment in real-world applications is challenging. This is due to limited access to internal workings and training data of the learned component. Furthermore, safety monitors should predict safety violations with low latency, while consuming a reasonable amount of computation. To address the challenges, we propose a safety monitoring method based on probabilistic time series forecasting. Given the learned component outputs and an operational context, we empirically investigate different Deep Learning (DL)-based probabilistic forecasting to predict the objective measure capturing the satisfaction or violation of a safety requirement (safety metric). We empirically evaluate safety metric and violation prediction accuracy, and inference latency and resource usage of four state-of-the-art models, with varying horizons, using autonomous aviation and autonomous driving case studies. Our results suggest that probabilistic forecasting of safety metrics, given learned component outputs and scenarios, is effective for safety monitoring. Furthermore, for both case studies, Temporal Fusion Transformer (TFT) was the most accurate model for predicting imminent safety violations, with acceptable latency and resource consumption.