Remarks on statistical aspects of safety analysis of complex systems

TL;DR

This paper explores statistical methods for safety testing of complex systems, emphasizing the importance of joint analysis of dependent outputs to avoid false safety conclusions.

Contribution

It introduces formulas for determining the number of simulation runs needed and compares statistical methods for analyzing multiple dependent output variables.

Findings

Different statistical methods require similar number of runs for safety assurance.

Testing dependent variables separately can lead to incorrect safety decisions.

Sign test and tolerance interval methods are recommended for joint analysis of dependent outputs.

Abstract

We analyze safety problems of complex systems using the methods of mathematical statistics for testing the output variables of a code simulating the operation of the system under consideration when the input variables are uncertain. We have defined a black box model of the code and derived formulas to calculate the number of runs needed for a given confidence level to achieve a preassigned measure of safety. In order to show the capabilities of different statistical methods, firstly we have investigated one output variable with unknown and known distribution functions. The general conclusion has been that the different methods do not bring about large differences in the number of runs needed to ensure a given level of safety. Analyzing the case of several statistically dependent output variables we have arrived at the conclusion that the testing of the variables separately may lead to false, safety related decisions with unforseen consequences. We have advised two methods: the sign test and the tolerance interval methods for testing more than one mutually dependent output variables.