AICO: Feature Significance Tests for Supervised Learning

TL;DR

AICO offers a scalable, statistically rigorous method for assessing feature importance in machine learning models without retraining or surrogate models, enhancing transparency and trust.

Contribution

It introduces a novel framework that provides exact p-values and confidence intervals for feature significance, applicable to large-scale models without additional training.

Findings

AICO reliably identifies influential features in real-world applications.

The method requires no retraining or surrogate modeling.

It offers finite-sample guarantees for feature importance testing.

Abstract

Machine learning is central to modern science, industry, and policy, yet its predictive power often comes at the cost of transparency: we rarely know which input features truly drive a model's predictions. Without such understanding, researchers cannot draw reliable conclusions, practitioners cannot ensure fairness or accountability, and policymakers cannot trust or govern model-based decisions. Existing tools for assessing feature influence are limited; most lack statistical guarantees, and many require costly retraining or surrogate modeling, making them impractical for large modern models. We introduce AICO, a broadly applicable framework that turns model interpretability into an efficient statistical exercise. AICO tests whether each feature genuinely improves predictive performance by masking its information and measuring the resulting change. The method provides exact, finite-sample feature p-values and confidence intervals for feature importance through a simple, non-asymptotic hypothesis testing procedure. It requires no retraining, surrogate modeling, or distributional assumptions, making it feasible for large-scale algorithms. In both controlled experiments and real applications, from credit scoring to mortgage-behavior prediction, AICO reliably identifies the variables that drive model behavior, providing a scalable and statistically principled path toward transparent and trustworthy machine learning.