Methodology for Testing and Evaluation of Safety Analytics Approaches

TL;DR

This paper introduces a simulation-based evaluation methodology for safety analytics approaches, enabling comprehensive, statistically robust comparisons and insights into their long-term effectiveness in safety risk assessment.

Contribution

The paper presents a novel simulation environment for systematic evaluation of safety analytics methods, overcoming limitations of historical data analysis.

Findings

Simulation-based evaluation reveals differences in approach performance.

Method provides statistically robust comparison of safety analytics.

Case study demonstrates practical utility of the methodology.

Abstract

There has been a significant increase in the development of data-driven safety analytics approaches in recent years. In light of these advances it has become imperative to evaluate such approaches in a principled way to determine their merits and limitations. To that end, we propose an evaluation methodology underpinned by a simulated environment that allows for a comprehensive assessment of safety analytics approaches. While assessing those approaches with historical field data is undoubtedly important, such an assessment has limited statistical power because it corresponds to only one realization. The proposed methodology enables validation over a large number of realizations, thereby circumventing the statistical limitations of evaluation on historical data. Moreover, by using a simulated environment one is able to clearly distinguish between the variability in the observed data and differences in performance between safety approaches. A simulated environment does this by comparing the approaches under controlled circumstances, resulting in a fair and systematic evaluation of the potential long-term benefits. We demonstrate the utility of the proposed methodology via a case study that compares a few candidate safety analytics approaches. These approaches differ in how they assimilate field safety data to assess safety risk and suggest mitigative actions. We show that the proposed methodology indeed reveals useful insights and quantifies the relative merits and drawbacks of the different approaches, which would be otherwise difficult to objectively determine in a real-world scenario.