Information-theoretic Quantification of High-order Feature Effects in Classification Problems

TL;DR

This paper introduces an information-theoretic method to quantify high-order feature interactions in classification models, improving interpretability by decomposing feature contributions into unique, synergistic, and redundant effects.

Contribution

It extends the Hi-Fi method using Conditional Mutual Information with a kNN estimator, enabling model-independent analysis of complex feature interactions.

Findings

Accurately recovers known interaction patterns in synthetic data.

Effectively analyzes gene expression data from TCGA-BRCA.

Provides detailed decomposition of feature effects.

Abstract

Understanding the contribution of individual features in predictive models remains a central goal in interpretable machine learning, and while many model-agnostic methods exist to estimate feature importance, they often fall short in capturing high-order interactions and disentangling overlapping contributions. In this work, we present an information-theoretic extension of the High-order interactions for Feature importance (Hi-Fi) method, leveraging Conditional Mutual Information (CMI) estimated via a k-Nearest Neighbor (kNN) approach working on mixed discrete and continuous random variables. Our framework decomposes feature contributions into unique, synergistic, and redundant components, offering a richer, model-independent understanding of their predictive roles. We validate the method using synthetic datasets with known Gaussian structures, where ground truth interaction patterns are analytically derived, and further test it on non-Gaussian and real-world gene expression data from TCGA-BRCA. Results indicate that the proposed estimator accurately recovers theoretical and expected findings, providing a potential use case for developing feature selection algorithms or model development based on interaction analysis.