Validation of Composite Systems by Discrepancy Propagation

TL;DR

This paper introduces a validation method that propagates bounds on distributional discrepancies through composite systems, enabling reliable failure probability estimation from simulations despite inaccuracies and data shifts.

Contribution

It develops a novel discrepancy propagation approach with convex relaxations to bound real system failure probabilities from imperfect simulations.

Findings

The method provides valid bounds for complex systems with realistic effects.

It effectively accounts for data shifts and model inaccuracies.

Demonstrates practical applicability in industrial validation scenarios.

Abstract

Assessing the validity of a real-world system with respect to given quality criteria is a common yet costly task in industrial applications due to the vast number of required real-world tests. Validating such systems by means of simulation offers a promising and less expensive alternative, but requires an assessment of the simulation accuracy and therefore end-to-end measurements. Additionally, covariate shifts between simulations and actual usage can cause difficulties for estimating the reliability of such systems. In this work, we present a validation method that propagates bounds on distributional discrepancy measures through a composite system, thereby allowing us to derive an upper bound on the failure probability of the real system from potentially inaccurate simulations. Each propagation step entails an optimization problem, where -- for measures such as maximum mean discrepancy (MMD) -- we develop tight convex relaxations based on semidefinite programs. We demonstrate that our propagation method yields valid and useful bounds for composite systems exhibiting a variety of realistic effects. In particular, we show that the proposed method can successfully account for data shifts within the experimental design as well as model inaccuracies within the simulation.