Size Matters: The Use and Misuse of Statistical Significance in Discrete Choice Models in the Transportation Academic Literature

TL;DR

This paper reviews transportation studies from 2014-2018 to highlight widespread misuse of statistical significance, emphasizing the need for better practices in interpreting model results and effect sizes.

Contribution

It provides an empirical assessment of statistical significance usage in transportation research and offers recommendations for improved analytical standards.

Findings

39% of studies relied solely on coefficient signs

67% did not differentiate statistical from practical significance

No studies considered statistical power

Abstract

In this paper we review the academic transportation literature published between 2014 and 2018 to evaluate where the field stands regarding the use and misuse of statistical significance in empirical analysis, with a focus on discrete choice models. Our results show that 39% of studies explained model results exclusively based on the sign of the coefficient, 67% of studies did not distinguish statistical significance from economic, policy or scientific significance in their conclusions, and none of the reviewed studies considered the statistical power of the tests. Based on these results we put forth a set of recommendations aimed at shifting the focus away from statistical significance towards proper and comprehensive assessment of effect magnitudes and other policy relevant quantities.