Statistical model uncertainty and OPERA-like time-of-flight measurements

TL;DR

This paper discusses the importance of addressing model uncertainty in time-of-flight measurements like OPERA, proposing strategies to improve statistical analysis and reduce errors in experimental conclusions.

Contribution

It introduces and evaluates strategies for managing model uncertainty in timing experiments, suggesting more effective methods for analyzing OPERA-like data.

Findings

Averaging probability distributions can mitigate model uncertainty.

Incorrect implementation of averaging may introduce errors.

Proposed alternative strategies could improve accuracy in timing measurements.

Abstract

Time-of-flight measurements such as the OPERA and MINOS experiments rely crucially on statistical analysis (as well as many other ingredients) for their conclusions. The nature of these experiments leads to a simple class of statistical models for the results; however, which model in the class is appropriate is not known exactly, as this depends on information obtained experimentally, which is subject to noise and other errors. To obtain robust conclusions, this problem, known as "model uncertainty," needs to be addressed, with quantitative bounds on the effect such uncertainty may have on the final result. The OPERA (and MINOS) analysis appears to take steps to mitigate the effects of model uncertainty, though without quantifying any remaining effect. We describe one of the strategies used (averaging individual probability distributions), and point out a potential source of error if this is not carried out correctly. We then argue that the correct version of this strategy is not the most effective, and suggest possible alternatives. These alternatives may give more accurate statistical results using the same data, allowing, for example, more accurate determination of the dependence of the anomalous time shift on energy. Which strategies work and how well can only be evaluated with access to the full data. Whether or not the anomalous result from OPERA turns out to be confirmed, we believe that techniques such as those presented here may be appropriate for the analysis of other timing experiments of this type.