Resampling NANCOVA: Nonparametric Analysis of Covariance in Small Samples

TL;DR

This paper introduces a resampling-based nonparametric ANCOVA method suitable for small samples, ordinal data, and multiple covariates, providing asymptotically exact tests with high power.

Contribution

It develops the first asymptotically exact resampling NANCOVA method that handles ordinal data, multiple covariates, and small samples without model assumptions.

Findings

Resampling NANCOVA maintains nominal type-I error in small samples.

It achieves up to 26% higher power than unadjusted tests.

The method is asymptotically exact and versatile for various data types.

Abstract

Analysis of covariance is a crucial method for improving precision of statistical tests for factor effects in randomized experiments. However, existing solutions suffer from one or more of the following limitations: (i) they are not suitable for ordinal data (as endpoints or explanatory variables); (ii) they require semiparametric model assumptions; (iii) they are inapplicable to small data scenarios due to often poor type-I error control; or (iv) they provide only approximate testing procedures and (asymptotically) exact test are missing. In this paper, we investigate a resampling approach to the NANCOVA framework, which is a fully nonparametric model based on relative effects that allows for an arbitrary number of covariates and groups, where both outcome variable (endpoint) and covariates can be metric or ordinal. Thereby, we evaluate novel NANCOVA tests and a nonparametric competitor test without covariate adjustment in extensive simulations. Unlike approximate tests in the NANCOVA framework, our resampling version showed good performance in small sample scenarios and maintained the nominal type-I error well. Resampling NANCOVA also provided consistently high power: up to 26% higher than the test without covariate adjustment in a small sample scenario with 4 groups and two covariates. Moreover, we prove that resampling NANCOVA provides an asymptotically exact testing procedure, which makes it the first one in the NANCOVA framework. In summary, resampling NANCOVA can be considered a viable tool for analysis of covariance that overcomes issues (i) - (iv).