Which Type of Statistical Uncertainty Helps Evidence-Based Policymaking? An Insight from a Survey Experiment in Ireland

TL;DR

This study compares how statistical significance and Bayesian probability information influence policymakers' perception of uncertainty, finding Bayesian probabilities better convey the continuous nature of uncertainty in decision-making.

Contribution

It provides experimental evidence that Bayesian probability information helps people better understand the continuous spectrum of uncertainty in policy contexts.

Findings

Bayesian probabilities improved perception of uncertainty continuity.

Significance testing led to more conservative policy adoption under high uncertainty.

Participants better understood uncertainty as a spectrum with Bayesian info.

Abstract

Which type of statistical uncertainty -- statistical (in)significance with a p-value, or a Bayesian probability -- enables people to see the continuous nature of uncertainty more clearly in a policymaking context? An original survey experiment used a hypothetical scenario, where participants from Ireland were asked whether to introduce a new bus line to reduce traffic jams, given a research report estimating its effectiveness. The treatments were uncertainty information: statistical significance with a p-value of 2%, statistical insignificance with a p-value of 25%, the 95% probability that the estimate is correct, and the 68% probability that the estimate is correct. In the case of lower uncertainty, both significance and Bayesian frameworks resulted in a large proportion of participants adopting the policy (0.82 and 0.91 respectively). In the case of higher uncertainty, the significance framework led a much smaller proportion of participants to adopt the policy (0.39 against 0.83). The findings suggest participants saw the continuous nature of uncertainty more clearly in the Bayesian framework than in the significance framework.