Generalized Permutation Framework for Testing Model Variable Significance

TL;DR

This paper introduces a flexible, non-parametric permutation framework for testing the significance of variables and their interactions in supervised machine learning models, especially effective for high-dimensional, low-sample datasets.

Contribution

It presents a novel permutation-based method that does not rely on distributional assumptions, enabling significance testing and interaction analysis of variables in complex models.

Findings

Successfully identified significant variables in Iris dataset

Detected interactions in RNA expression data

Framework applicable to high-dimensional datasets

Abstract

A common problem in machine learning is determining if a variable significantly contributes to a model's prediction performance. This problem is aggravated for datasets, such as gene expression datasets, that suffer the worst case of dimensionality: a low number of observations along with a high number of possible explanatory variables. In such scenarios, traditional methods for testing variable statistical significance or constructing variable confidence intervals do not apply. To address these problems, we developed a novel permutation framework for testing the significance of variables in supervised models. Our permutation framework has three main advantages. First, it is non-parametric and does not rely on distributional assumptions or asymptotic results. Second, it not only ranks model variables in terms of relative importance, but also tests for statistical significance of each variable. Third, it can test for the significance of the interaction between model variables. We applied this permutation framework to multi-class classification of the Iris flower dataset and of brain regions in RNA expression data, and using this framework showed variable-level statistical significance and interactions.