One optional observation inflates $\alpha$ by $100/\sqrt{n}$ per cent

TL;DR

This paper derives an asymptotic formula showing how an additional observation can inflate the significance level of one-sample tests by approximately 100 divided by the square root of the sample size, highlighting potential issues in sequential testing.

Contribution

It provides a new asymptotic formula for the actual level of tests when considering multiple testing stages, especially for the case of one additional observation.

Findings

An optional observation inflates the test level by about 100/√n percent.

The asymptotic formula applies to standard tests at usual significance levels.

A simple proof is provided for the case of a single additional test with Gaussian data.

Abstract

For one-sample level $\alpha$ tests $\psi_m$ based on independent observations $X_1,...,X_m$, we prove an asymptotic formula for the actual level of the test rejecting if at least one of the tests $\psi_{n},...,\psi_{n+k}$ would reject. For $k=1$ and usual tests at usual levels $\alpha$, the result is approximately summarized by the title of this paper. Our method of proof, relying on some second order asymptotic statistics as developed by Pfanzagl and Wefelmeyer, might also be useful for proper sequential analysis. A simple and elementary alternative proof is given for $k=1$ in the special case of the Gauss test.