Statistical significance in choice modelling: computation, usage and reporting

TL;DR

This paper critically examines the use of statistical significance in choice modelling, highlighting issues in computation, interpretation, and reporting, and emphasizing the importance of behavioral and policy relevance.

Contribution

It provides a detailed review of uncertainty measures, discusses common misunderstandings, and offers guidance on proper reporting and interpretation in choice modelling research.

Findings

Over-reliance on 95% confidence levels and misunderstandings of significance.

Lack of precision in reporting uncertainty measures, especially p-values and star measures.

Need to consider behavioral and policy significance alongside statistical significance.

Abstract

This paper offers a commentary on the use of notions of statistical significance in choice modelling. We review the reasons for uncertainty in parameter estimates, provide a precise discussion on the computation of measures of uncertainty and confidence intervals, and discuss the use of statistical tests. We argue that, as in many other areas of science, there is an over-reliance on 95\% confidence levels, and misunderstandings of the meaning of significance. We also observe a lack of precision in the reporting of measures of uncertainty in many studies, especially when using $p$-values and even more so with \emph{star} measures. The paper also stresses the importance of considering behavioural or policy significance in addition to statistical significance. Finally, we stress a number of points that are specific to choice modelling and which require special attention, notably in relation to derived measures such as willingness-to-pay, the treatment of random heterogeneity, and the use of repeated choice data.