PERMUTOOLS: A MATLAB Package for Multivariate Permutation Testing

TL;DR

PERMUTOOLS is a MATLAB package that facilitates multivariate permutation testing and effect size measurement, offering a distribution-free alternative to traditional parametric methods, with advantages in multiple comparison correction and confidence interval estimation.

Contribution

This paper introduces PERMUTOOLS, a MATLAB package that implements efficient multivariate permutation testing and effect size measurement, filling a gap in available software tools.

Findings

Provides a MATLAB tool for permutation testing

Enables empirical confidence interval estimation

Supports multiple comparison correction

Abstract

Statistical hypothesis testing and effect size measurement are routine parts of quantitative research. Advancements in computer processing power have greatly improved the capability of statistical inference through the availability of resampling methods. However, many of the statistical practices used today are based on traditional, parametric methods that rely on assumptions about the underlying population. These assumptions may not always be valid, leading to inaccurate results and misleading interpretations. Permutation testing, on the other hand, generates the sampling distribution empirically by permuting the observed data, providing distribution-free hypothesis testing. Furthermore, this approach lends itself to a powerful method for multiple comparison correction - known as max correction - which is less prone to type II errors than conventional correction methods. Parametric methods have also traditionally been utilized for estimating the confidence interval of various test statistics and effect size measures. However, these too can be estimated empirically using permutation or bootstrapping techniques. Whilst resampling methods are generally considered preferable, many popular programming languages and statistical software packages lack efficient implementations. Here, we introduce PERMUTOOLS, a MATLAB package for multivariate permutation testing and effect size measurement.