The Exchangeability Assumption for Permutation Tests of Multiple Regression Models: Implications for Statistics and Data Science Educators

TL;DR

This paper examines the exchangeability assumption underlying permutation tests in multiple linear regression, discussing its implications, limitations, and pedagogical approaches for teaching resampling methods in statistics education.

Contribution

It clarifies the role of exchangeability in permutation tests for regression, evaluates different permutation schemes, and offers educational recommendations for instructors.

Findings

Most permutation schemes yield similar Type I error rates.

Violations like improper clustering can affect error rates.

Understanding exchangeability enhances statistical inference education.

Abstract

Permutation tests are a powerful and flexible approach to inference via resampling. As computational methods become more ubiquitous in the statistics curriculum, use of permutation tests has become more tractable. At the heart of the permutation approach is the exchangeability assumption, which determines the appropriate null sampling distribution. We explore the exchangeability assumption in the context of permutation tests for multiple linear regression models, including settings where the assumption is not tenable. Various permutation schemes for the multiple linear regression setting have been proposed and assessed in the literature. As has been demonstrated previously, in most settings, the choice of how to permute a multiple linear regression model does not materially change inferential conclusions with respect to Type I errors. However, some violations (e.g., when clustering is not appropriately accounted for) lead to issues with Type I error rates. Regardless, we believe that understanding (1) exchangeability in the multiple linear regression setting and also (2) how it relates to the null hypothesis of interest is valuable. We close with pedagogical recommendations for instructors who want to bring multiple linear regression permutation inference into their classroom as a way to deepen student understanding of resampling-based inference.