A robust and p-hacking-proof significance test under variance uncertainty

TL;DR

This paper introduces a new significance testing method that is robust against p-hacking and variance uncertainty, ensuring reliable results even with manipulated or variably distributed data.

Contribution

It develops a novel p-hacking-proof significance test using sublinear expectation theory to handle variance bounds and data manipulation.

Findings

Effectively controls type I error under variance uncertainty

Maintains satisfactory statistical power

Outperforms traditional tests in simulations

Abstract

P-hacking poses challenges to traditional hypothesis testing. In this paper, we propose a robust method for the one-sample significance test that can protect against p-hacking from sample manipulation. Precisely, assuming a sequential arrival of the data whose variance can be time-varying and for which only lower and upper bounds are assumed to exist with possibly unknown values, we use the modern theory of sublinear expectation to build a testing procedure which is robust under such variance uncertainty, and can protect the significance level against potential data manipulation by an experimenter. It is shown that our new method can effectively control the type I error while preserving a satisfactory power, yet a traditional rejection criterion performs poorly under such variance uncertainty. Our theoretical results are well confirmed by a detailed simulation study.