The nonparametric Behrens-Fisher problem in small samples

TL;DR

This paper reviews variance estimation methods for the Wilcoxon-Mann-Whitney test, especially in small samples, and evaluates their performance through simulations to recommend the most reliable approach.

Contribution

It provides a comprehensive comparison of variance estimators and distribution approximations for the Wilcoxon-Mann-Whitney test in small samples, highlighting the best practices.

Findings

Performance depends on heteroskedasticity and sample allocation.

t-approximation with Perme and Manevski's estimator controls type I error well.

Different estimators vary in accuracy depending on sample conditions.

Abstract

While there appears to be a general consensus in the literature on the definition of the estimand and estimator associated with the Wilcoxon-Mann-Whitney test, it seems somewhat less clear as to how best to estimate the variance. In addition to the Wilcoxon-Mann-Whitney test, we review different proposals of variance estimators consistent under both the null hypothesis and the alternative. Moreover, in case of small sample sizes, an approximation of the distribution of the test statistic based on the t-distribution, a logit transformation and a permutation approach have been proposed. Focussing as well on different estimators of the degrees of freedom as regards the t-approximation, we carried out simulations for a range of scenarios, with results indicating that the performance of different variance estimators in terms of controlling the type I error rate largely depends on the heteroskedasticity pattern and the sample size allocation ratio, not on the specific type of distributions employed. By and large, a particular t-approximation together with Perme and Manevski's variance estimator best maintains the nominal significance level