Policy Implications of Statistical Estimates: A General Bayesian Decision-Theoretic Model for Binary Outcomes

TL;DR

This paper introduces a Bayesian decision-theoretic model for evaluating policy effects on binary outcomes, addressing limitations of significance testing by incorporating uncertainty, effect size variation, and substantive significance.

Contribution

It proposes a novel causal binary loss function model that improves policy evaluation by integrating effect size ranges, probabilities, and costs, with practical applications and an R package.

Findings

The model effectively compares expected losses under different policies.

Applications demonstrate the model's practical utility.

Provides an R package for implementation.

Abstract

How should we evaluate the effect of a policy on the likelihood of an undesirable event, such as conflict? The significance test has three limitations. First, relying on statistical significance misses the fact that uncertainty is a continuous scale. Second, focusing on a standard point estimate overlooks the variation in plausible effect sizes. Third, the criterion of substantive significance is rarely explained or justified. A new Bayesian decision-theoretic model, "causal binary loss function model," overcomes these issues. It compares the expected loss under a policy intervention with the one under no intervention. These losses are computed based on a particular range of the effect sizes of a policy, the probability mass of this effect size range, the cost of the policy, and the cost of the undesirable event the policy intends to address. The model is more applicable than common statistical decision-theoretic models using the standard loss functions or capturing costs in terms of false positives and false negatives. I exemplify the model's use through three applications and provide an R package.