Adaptative significance levels in normal mean hypothesis testing

TL;DR

This paper proposes a method to adaptively determine significance levels in normal mean hypothesis testing using the FBST, accounting for sample size to improve decision accuracy.

Contribution

It introduces a sample size-dependent cutoff for the FBST evidence measure, enhancing hypothesis testing by balancing error probabilities.

Findings

The cutoff varies with sample size to control error rates.

The method minimizes a linear combination of type I and II errors.

Improves decision-making in Bayesian hypothesis testing.

Abstract

The Full Bayesian Significance Test (FBST) for precise hypotheses was presented by Pereira and Stern (1999) as a Bayesian alternative instead of the traditional significance test based on p-value. The FBST uses the evidence in favor of the null hypothesis ($H_0$) calculated as the complement of the posterior probability of the highest posterior density region, which is tangent to the set defined by $H_0$. An important practical issue for the implementation of the FBST is the determination of how large the evidence must be in order to decide for its rejection. In the Classical significance tests, the most used measure for rejecting a hypothesis is p-value. It is known that p-value decreases as sample size increases, so by setting a single significance level, it usually leads $H_0$ rejection. In the FBST procedure, the evidence in favor of $H_0$ exhibits the same behavior as the p-value when the sample size increases. This suggests that the cut-off point to define the rejection of $H_0$ in the FBST should be a sample size function. In this work, we focus on the case of two-sided normal mean hypothesis testing and present a method to find a cut-off value for the evidence in the FBST by minimizing the linear combination of the type I error probability and the expected type II error probability for a given sample size.