Runtime Monitoring of Perception-Based Autonomous Systems via Embedding Temporal Logic

TL;DR

This paper introduces Embedding Temporal Logic (ETL), a novel runtime monitoring approach for perception-based autonomous systems that operates directly in learned embedding spaces to specify and evaluate high-level perceptual behaviors.

Contribution

ETL enables specification and monitoring of perceptual concepts directly in embedding spaces, bypassing traditional logical propositions and enhancing semantic expressiveness.

Findings

ETL accurately monitors temporally extended perceptual behaviors.

ETL shows strong empirical agreement with ground-truth semantics.

The approach is effective across multiple manipulation environments.

Abstract

Runtime monitoring of autonomous systems traditionally relies on mapping continuous sensor observations to discrete logical propositions defined over low-dimensional state variables. This abstraction breaks down in perception-driven settings, where such mappings require additional learned modules that are often computationally expensive, brittle, and semantically misaligned. In this work, we propose Embedding Temporal Logic (ETL), a temporal logic that performs monitoring directly in learned embedding spaces. ETL defines predicates through distances between observed embeddings and target embeddings derived from reference observations. This formulation allows specifications to capture high-level perceptual concepts, such as similarity to visual goals or avoidance of semantic regions, that are difficult or impossible to express using traditional predicates. By composing these predicates with temporal operators, ETL naturally expresses temporally extended and sequential perceptual behaviors. We introduce ETL monitors for evaluating specifications over bounded embedding traces, along with a conformal calibration procedure that provides reliable and safety-oriented predicate evaluation. We evaluate our approach across multiple manipulation environments to show that ETL achieves strong empirical agreement with ground-truth semantics, including accurate monitoring of temporally composed behaviors.