Causal Entropy and Information Gain for Measuring Causal Control

TL;DR

This paper introduces causal entropy and causal information gain, new information-theoretic measures that incorporate causal structure to evaluate feature importance and control over outcomes, improving interpretability in AI models.

Contribution

It proposes causal versions of entropy and mutual information that explicitly account for causality, enabling better feature selection based on causal influence.

Findings

Causal information gain outperforms standard mutual information in identifying causally relevant features.

Fundamental links between causal quantities and causal effects are established.

Demonstrated the effectiveness of causal information gain in feature selection tasks.

Abstract

Artificial intelligence models and methods commonly lack causal interpretability. Despite the advancements in interpretable machine learning (IML) methods, they frequently assign importance to features which lack causal influence on the outcome variable. Selecting causally relevant features among those identified as relevant by these methods, or even before model training, would offer a solution. Feature selection methods utilizing information theoretical quantities have been successful in identifying statistically relevant features. However, the information theoretical quantities they are based on do not incorporate causality, rendering them unsuitable for such scenarios. To address this challenge, this article proposes information theoretical quantities that incorporate the causal structure of the system, which can be used to evaluate causal importance of features for some given outcome variable. Specifically, we introduce causal versions of entropy and mutual information, termed causal entropy and causal information gain, which are designed to assess how much control a feature provides over the outcome variable. These newly defined quantities capture changes in the entropy of a variable resulting from interventions on other variables. Fundamental results connecting these quantities to the existence of causal effects are derived. The use of causal information gain in feature selection is demonstrated, highlighting its superiority over standard mutual information in revealing which features provide control over a chosen outcome variable. Our investigation paves the way for the development of methods with improved interpretability in domains involving causation.