Vision-Based Runtime Monitoring under Varying Specifications using Semantic Latent Representations

TL;DR

This paper introduces a semantic latent representation-based approach for certified runtime monitoring of past-time signal temporal logic from visual data, ensuring guarantees under partial observability.

Contribution

It proves the semantic basis as the minimal prediction target for monotone, 1-Lipschitz reusable interfaces and develops a rolling prediction monitor for online inference.

Findings

Semantic basis provides tight bounds at long horizons.

Rolling monitor performs better at short horizons.

Both methods satisfy empirical conformal coverage guarantees.

Abstract

We study certified runtime monitoring of past-time signal temporal logic (ptSTL) from visual observations under partial observability. The monitor must infer safety-relevant quantities from images and provide finite-sample guarantees, while being \emph{reusable}: once trained and calibrated, it should certify any formula in a target fragment without per-formula retraining. For fragments induced by a finite dictionary of temporal atoms, we prove that the \emph{semantic basis}, the vector of atom robustness scores, is the minimum prediction target within the class of monotone, 1-Lipschitz reusable interfaces: any formula is evaluated by a deterministic decoder derived from the parse tree, and a single conformal calibration pass certifies the entire fragment with no union bound. We also introduce a \emph{rolling prediction monitor} that predicts only current predicate values and reconstructs temporal history online; this is easier to learn but grows conservative at long horizons. On a pedestrian-crossroad benchmark, rolling achieves tighter certified bounds at short horizons while the semantic-basis monitor is up to 4-times tighter at long horizons. We validate the presented monitors on real-world Waymo driving data, where both monitors satisfy the conformal coverage guarantee empirically.