# Comparing researchers’ degree of dichotomous thinking using frequentist versus Bayesian null hypothesis testing

## TL;DR

This study compares how researchers interpret statistical evidence using p-values and Bayes factors, finding that p-values lead to more abrupt shifts in belief around thresholds.

## Contribution

The study introduces a novel comparison of cliff effects in statistical interpretation between frequentist and Bayesian approaches.

## Key findings

- A higher proportion of cliff effects was observed in p-value conditions compared to Bayes factor conditions.
- No clear evidence was found that presentation mode affects the proportion of cliff effects.
- The study highlights differences in how statistical evidence is interpreted using p-values versus Bayes factors.

## Abstract

A large amount of scientific literature in social and behavioural sciences bases their conclusions on one or more hypothesis tests. As such, it is important to obtain more knowledge about how researchers in social and behavioural sciences interpret quantities that result from hypothesis test metrics, such as p-values and Bayes factors. In the present study, we explored the relationship between obtained statistical evidence and the degree of belief or confidence that there is a positive effect in the population of interest. In particular, we were interested in the existence of a so-called cliff effect: A qualitative drop in the degree of belief that there is a positive effect around certain threshold values of statistical evidence (e.g., at p = 0.05). We compared this relationship for p-values to the relationship for corresponding degrees of evidence quantified through Bayes factors, and we examined whether this relationship was affected by two different modes of presentation (in one mode the functional form of the relationship across values was implicit to the participant, whereas in the other mode it was explicit). We found evidence for a higher proportion of cliff effects in p-value conditions than in BF conditions (N = 139), but we did not get a clear indication whether presentation mode had an effect on the proportion of cliff effects.

The stage 1 protocol for this Registered Report was accepted in principle on 2 June 2023. The protocol, as accepted by the journal, can be found at: https://doi.org/10.17605/OSF.IO/5CW6P.