Conformance Testing for Stochastic Cyber-Physical Systems

TL;DR

This paper introduces probabilistic measures for assessing the conformance of stochastic dynamical systems, enabling efficient comparison of models and real systems using data-driven statistical tools.

Contribution

It proposes a novel probabilistic framework for stochastic conformance, including non-conformance risk, with empirical validation on diverse real-world systems.

Findings

Probabilistic conformance measures effectively compare stochastic systems.

Non-conformance risk quantifies the likelihood of systems not conforming.

Data-driven estimation methods enable practical application of the framework.

Abstract

Conformance is defined as a measure of distance between the behaviors of two dynamical systems. The notion of conformance can accelerate system design when models of varying fidelities are available on which analysis and control design can be done more efficiently. Ultimately, conformance can capture distance between design models and their real implementations and thus aid in robust system design. In this paper, we are interested in the conformance of stochastic dynamical systems. We argue that probabilistic reasoning over the distribution of distances between model trajectories is a good measure for stochastic conformance. Additionally, we propose the non-conformance risk to reason about the risk of stochastic systems not being conformant. We show that both notions have the desirable transference property, meaning that conformant systems satisfy similar system specifications, i.e., if the first model satisfies a desirable specification, the second model will satisfy (nearly) the same specification. Lastly, we propose how stochastic conformance and the non-conformance risk can be estimated from data using statistical tools such as conformal prediction. We present empirical evaluations of our method on an F-16 aircraft, an autonomous vehicle, a spacecraft, and Dubin's vehicle.