A Comparison of the Bayesian Posterior Probability and the Frequentist $p$-Value in Testing Equivalence Hypotheses

TL;DR

This paper compares Bayesian posterior probability and frequentist p-values in equivalence testing, proposing a Bayesian approach that can be more powerful and less conservative under certain conditions, with analysis and simulations demonstrating their performance.

Contribution

It introduces a Bayesian equivalence test using posterior probabilities within the TOST framework and compares its performance to traditional p-value based tests.

Findings

Bayesian posterior probabilities can be more powerful than p-values under specific prior settings.

The correlation between Bayesian and frequentist measures of evidence is derived.

The Bayesian approach shows better control of type I error and higher power in simulations.

Abstract

Equivalence tests, otherwise known as parity or similarity tests, are frequently used in ``bioequivalence studies" to establish practical equivalence rather than the usual statistical significant difference. In this article, we propose an equivalence test using both the $p$-value and a Bayesian procedure by computing the posterior probability that the null hypothesis is true. Since these posterior probabilities follow the uniform $[0,1]$ distribution under the null hypothesis, we use them in a Two One-Sided Test (TOST) procedure to perform equivalence tests. For certain specifications of the prior parameters, test based on these posterior probabilities are more powerful and less conservative than those based on the $p$-value. We compare the parameter values that maximize the power functions of tests based on these two measures of evidence when using different equivalence margins. We also derive the correlation coefficient between these two measures of evidence. Furthermore, we also consider the effect of the prior variance on the conservativity and power function of the test based on the posterior probabilities. Finally, we provide examples and a small-scale simulation study to compare their performance in terms of type I error rate control and power in a single test, as well as in multiple testing, considering the power of the false discovery rate procedure.