Problems with information theoretic approaches to causal learning

TL;DR

This paper critiques the use of traditional information theoretic measures in causal learning, highlighting their limitations with finite data and proposing data compression methods as more robust alternatives.

Contribution

It demonstrates the pitfalls of infotheoretic estimation in causal learning and advocates for data compression approaches to improve robustness and accuracy.

Findings

Infotheoretic measures can be unreliable with short sequences.

Data compression methods are more robust for causality testing.

Conditioning processes can increase confusion, complicating causal modeling.

Abstract

The language of information theory is favored in both causal reasoning and machine learning frameworks. But, is there a better language than this? In this study, we demonstrate the pitfalls of infotheoretic estimation using first order statistics on (short) sequences for causal learning. We recommend the use of data compression based approaches for causality testing since these make very little assumptions on data as opposed to infotheoretic measures, and are more robust to finite data length effects. We conclude with a discussion on the challenges posed in modeling the effects of conditioning process $X$ with another process $Y$ in causal machine learning. Specifically, conditioning can increase 'confusion' which can be difficult to model by classical information theory. A conscious causal agent creates new choices, decisions and meaning which poses huge challenges for AI.