On the T-test

TL;DR

This paper critically examines the limitations of the traditional T-test, demonstrating its non-uniform applicability across bounded distributions and proposing a generalized version to address these issues.

Contribution

It provides the first proof that the T-test's approximation does not hold uniformly and introduces a generalized test accommodating distribution variability.

Findings

Normal and Student's approximations fail uniformly for bounded distributions.

Lower bounds on approximation errors are established.

A generalized T-test is proposed to improve robustness.

Abstract

The $T$-test is probably the most popular statistical test; it is routinely recommended by the textbooks. The applicability of the test relies upon the validity of normal or Student's approximation to the distribution of Student's statistic $\,t_n$. However, the latter assumption is not valid as often as assumed. We show that normal or Student's approximation to $\,\L(t_n)\,$ does not hold uniformly even in the class $\,{\cal P}_n\,$ of samples from zero-mean unit-variance bounded distributions. We present lower bounds to the corresponding error. The fact that a non-parametric test is not applicable uniformly to samples from the class $\,{\cal P}_n\,$ seems to be established for the first time. It means the $T$-test can be misleading, and should not be recommended in its present form. We suggest a generalisation of the test that allows for variability of possible limiting/approximating distributions to $\,\L(t_n)$.