Post-hoc Interpretability Illumination for Scientific Interaction Discovery

TL;DR

This paper introduces a novel post-hoc interpretability method called Iterative Kings' Forests (iKF) that uncovers complex variable interactions, enhancing explainability and scientific discovery in decision-making models.

Contribution

The paper presents iKF, a new iterative method for identifying multi-order variable interactions and classifying interaction types, improving interpretability over existing tools.

Findings

iKF effectively uncovers complex variable interactions.

iKF provides meaningful inference metrics for interaction analysis.

Experimental results show iKF's strong interpretive power across scientific fields.

Abstract

Model interpretability and explainability have garnered substantial attention in recent years, particularly in decision-making applications. However, existing interpretability tools often fall short in delivering satisfactory performance due to limited capabilities or efficiency issues. To address these challenges, we propose a novel post-hoc method: Iterative Kings' Forests (iKF), designed to uncover complex multi-order interactions among variables. iKF iteratively selects the next most important variable, the "King", and constructs King's Forests by placing it at the root node of each tree to identify variables that interact with the "King". It then generates ranked short lists of important variables and interactions of varying orders. Additionally, iKF provides inference metrics to analyze the patterns of the selected interactions and classify them into one of three interaction types: Accompanied Interaction, Synergistic Interaction, and Hierarchical Interaction. Extensive experiments demonstrate the strong interpretive power of our proposed iKF, highlighting its great potential for explainable modeling and scientific discovery across diverse scientific fields.