How to analyze data in a factorial design? An extensive simulation study

TL;DR

This paper compares various statistical methods for analyzing factorial design data through extensive simulations, focusing on their assumptions, error rates, and power, especially under non-standard conditions and different significance levels.

Contribution

It provides a comprehensive simulation-based comparison of methods for factorial data analysis, highlighting their strengths and limitations under various assumptions.

Findings

Certain methods maintain control of type-I error under assumption violations.

Some approaches offer higher power in small sample scenarios.

Performance varies significantly depending on data distribution and sample size.

Abstract

Factorial designs are frequently used in different fields of science, e.g. psychological, medical or biometric studies. Standard approaches, as the ANOVA $F$-test, make different assumptions on the distribution of the error terms, the variances or the sample sizes in the different groups. Because of time constraints or a lack of statistical background, many users do not check these assumptions; enhancing the risk of potentially inflated type-$I$ error rates or a substantial loss of power. It is the aim of the present paper, to give an overview of different methods without such restrictive assumptions and to identify situations in which one method is superior compared to others. In particular, after summarizing their underlying assumptions, the different approaches are compared within extensive simulations. To also address the current discussion about redefining the statistical significance level, we also included simulations for the 0.5\% level.