Querying Labeled Time Series Data with Scenario Programs

TL;DR

This paper introduces a formal method and algorithm for querying labeled time series sensor data to identify scenarios matching abstract representations, improving accuracy and speed over existing large language models.

Contribution

It presents a novel approach using Scenic scenario programs to formally define and efficiently query matching failure scenarios in real-world sensor datasets.

Findings

Algorithm outperforms commercial vision LLMs in accuracy

Querying is significantly faster and scalable

Effective validation of simulation scenarios in real data

Abstract

Simulation-based testing has become a crucial complement to road testing for ensuring the safety of cyber physical systems (CPS). As a result, significant research efforts have been directed toward identifying failure scenarios within simulation environments. However, a critical question remains. Are the AV failure scenarios discovered in simulation reproducible on actual systems in the real world? The sim-to-real gap caused by differences between simulated and real sensor data means that failure scenarios identified in simulation might either be artifacts of synthetic sensor data or actual issues that also occur with real sensor data. To address this, an effective approach to validating simulated failure scenarios is to locate occurrences of these scenarios within real-world datasets and verify whether the failure persists on the datasets. To this end, we introduce a formal definition of how labeled time series sensor data can match an abstract scenario, represented as a scenario program using the Scenic probabilistic programming language. We present a querying algorithm that, given a scenario program and a labeled dataset, identifies the subset of data that matches the specified scenario. Our experiment shows that our algorithm is more accurate and orders of magnitude faster in querying scenarios than the state-of-the-art commercial vision large language models, and can scale with the duration of queried time series data.