Neural Predictive Monitoring under Partial Observability

TL;DR

This paper introduces a neural predictive monitoring method that maintains accurate and reliable system safety predictions under partial and noisy observations by combining deep learning, conformal prediction, and active learning techniques.

Contribution

It extends Neural Predictive Monitoring to handle partial observability using two approaches and incorporates conformal prediction for uncertainty quantification and active learning.

Findings

Achieves highly accurate reachability predictions under partial observability.

Provides tight prediction regions with guaranteed coverage.

Enables active learning to improve monitor accuracy over time.

Abstract

We consider the problem of predictive monitoring (PM), i.e., predicting at runtime future violations of a system from the current state. We work under the most realistic settings where only partial and noisy observations of the state are available at runtime. Such settings directly affect the accuracy and reliability of the reachability predictions, jeopardizing the safety of the system. In this work, we present a learning-based method for PM that produces accurate and reliable reachability predictions despite partial observability (PO). We build on Neural Predictive Monitoring (NPM), a PM method that uses deep neural networks for approximating hybrid systems reachability, and extend it to the PO case. We propose and compare two solutions, an end-to-end approach, which directly operates on the rough observations, and a two-step approach, which introduces an intermediate state estimation step. Both solutions rely on conformal prediction to provide 1) probabilistic guarantees in the form of prediction regions and 2) sound estimates of predictive uncertainty. We use the latter to identify unreliable (and likely erroneous) predictions and to retrain and improve the monitors on these uncertain inputs (i.e., active learning). Our method results in highly accurate reachability predictions and error detection, as well as tight prediction regions with guaranteed coverage.