A controlled experiment for the empirical evaluation of safety analysis techniques for safety-critical software

TL;DR

This study empirically compares FTA, FMEA, and STPA safety analysis techniques for safety-critical software, finding STPA more effective in identifying safety requirements but more time-consuming.

Contribution

It provides a quantitative comparison of three safety analysis techniques, highlighting STPA's strengths and trade-offs in practical application.

Findings

STPA identifies more safety requirements than FTA and FMEA.

No significant difference in applicability, understandability, ease of use.

STPA requires more time to perform, especially for inexperienced analysts.

Abstract

Context: Today's safety critical systems are increasingly reliant on software. Software becomes responsible for most of the critical functions of systems. Many different safety analysis techniques have been developed to identify hazards of systems. FTA and FMEA are most commonly used by safety analysts. Recently, STPA has been proposed with the goal to better cope with complex systems including software. Objective: This research aimed at comparing quantitatively these three safety analysis techniques with regard to their effectiveness, applicability, understandability, ease of use and efficiency in identifying software safety requirements at the system level. Method: We conducted a controlled experiment with 21 master and bachelor students applying these three techniques to three safety-critical systems: train door control, anti-lock braking and traffic collision and avoidance. Results: The results showed that there is no statistically significant difference between these techniques in terms of applicability, understandability and ease of use, but a significant difference in terms of effectiveness and efficiency is obtained. Conclusion: We conclude that STPA seems to be an effective method to identify software safety requirements at the system level. In particular, STPA addresses more different software safety requirements than the traditional techniques FTA and FMEA, but STPA needs more time to carry out by safety analysts with little or no prior experience.