Logical contradictions in the One-way ANOVA and Tukey-Kramer multiple comparisons tests with more than two groups of observations

TL;DR

This paper reveals logical contradictions between One-way ANOVA and Tukey-Kramer tests for more than two groups, demonstrating that they can give conflicting results even under standard assumptions, and extends the analysis to multivariable linear regression.

Contribution

It constructs explicit examples showing the inconsistency between ANOVA and Tukey-Kramer tests for multiple groups, challenging their assumed equivalence under standard conditions.

Findings

ANOVA and Tukey-Kramer agree for two groups.

Contradictions arise for more than two groups under INAH assumptions.

Similar contradictions are shown for multivariable linear regression.

Abstract

We show that the One-way ANOVA and Tukey-Kramer (TK) tests agree on any sample with two groups. This result is based on a simple identity connecting the Fisher-Snedecor and studentized probabilistic distributions and is proven without any additional assumptions; in particular, the standard ANOVA assumptions (independence, normality, and homoscedasticity (INAH)) are not needed. In contrast, it is known that for a sample with k > 2 groups of observations, even under the INAH assumptions, with the same significance level $\alpha$, the above two tests may give opposite results: (i) ANOVA rejects its null hypothesis $H_0^{A}: \mu_1 = \ldots = \mu_k$, while the TK one, $H_0^{TK}(i,j): \mu_i = \mu_j$, is not rejected for any pair $i, j \in \{1, \ldots, k\}$; (ii) the TK test rejects $H_0^{TK}(i,j)$ for a pair $(i, j)$ (with $i \neq j$) while ANOVA does not reject $H_0^{A}$. We construct two large infinite pseudo-random families of samples of both types satisfying INAH: in case (i) for any $k \geq 3$ and in case (ii) for some larger $k$. Furthermore, in case (ii) ANOVA, being restricted to the pair of groups $(i,j)$, may reject equality $\mu_i = \mu_j$ with the same $\alpha$. This is an obvious contradiction, since $\mu_1 = \ldots = \mu_k$ implies $\mu_i = \mu_j$ for all $i, j \in \{1, \ldots, k\}.$ Similar contradictory examples are constructed for the Multivariable Linear Regression (MLR). However, for these constructions it seems difficult to verify the Gauss-Markov assumptions, which are standardly required for MLR.