An Aligned Rank Transform Procedure for Multifactor Contrast Tests

TL;DR

This paper introduces ART-C, a new algorithm for contrast testing in nonparametric multifactor experiments, which improves accuracy and power over existing methods, validated on extensive simulated data.

Contribution

We developed and validated ART-C, a novel contrast testing algorithm within the ART framework, addressing limitations of previous methods and extending ARTool for broader use.

Findings

ART-C maintains correct Type I error rates.

ART-C has higher statistical power than traditional tests.

Validated on 72,000 simulated data sets.

Abstract

Data from multifactor HCI experiments often violates the normality assumption of parametric tests (i.e., nonconforming data). The Aligned Rank Transform (ART) is a popular nonparametric analysis technique that can find main and interaction effects in nonconforming data, but leads to incorrect results when used to conduct contrast tests. We created a new algorithm called ART-C for conducting contrasts within the ART paradigm and validated it on 72,000 data sets. Our results indicate that ART-C does not inflate Type I error rates, unlike contrasts based on ART, and that ART-C has more statistical power than a t-test, Mann-Whitney U test, Wilcoxon signed-rank test, and ART. We also extended a tool called ARTool with our ART-C algorithm for both Windows and R. Our validation had some limitations (e.g., only six distribution types, no mixed factorial designs, no random slopes), and data drawn from Cauchy distributions should not be analyzed with ART-C.