iLOCO: Distribution-Free Inference for Feature Interactions

TL;DR

This paper introduces iLOCO, a model-agnostic, distribution-free method for statistically inferring the importance of feature interactions, addressing a gap in interpretability tools for complex models.

Contribution

We propose the first distribution-free inference method for feature interactions, including a new interaction importance metric and efficient confidence interval computation.

Findings

iLOCO outperforms existing interaction importance methods

Provides the first statistical inference approach for feature interactions

Efficient ensemble method enhances computational feasibility

Abstract

Feature importance measures are widely studied and are essential for understanding model behavior, guiding feature selection, and enhancing interpretability. However, many machine learning fitted models involve complex interactions between features. Existing feature importance metrics fail to capture these pairwise or higher-order effects, while existing interaction metrics often suffer from limited applicability or excessive computation; no methods exist to conduct statistical inference for feature interactions. To bridge this gap, we first propose a new model-agnostic metric, interaction Leave-One-Covariate-Out (iLOCO), for measuring the importance of pairwise feature interactions, with extensions to higher-order interactions. Next, we leverage recent advances in LOCO inference to develop distribution-free and assumption-light confidence intervals for our iLOCO metric. To address computational challenges, we also introduce an ensemble learning method for calculating the iLOCO metric and confidence intervals that we show is both computationally and statistically efficient. We validate our iLOCO metric and our confidence intervals on both synthetic and real data sets, showing that our approach outperforms existing methods and provides the first inferential approach to detecting feature interactions.