De-paradox Tree: Breaking Down Simpson's Paradox via A Kernel-Based Partition Algorithm

TL;DR

De-paradox Tree is an interpretable algorithm that detects and explains Simpson's paradox in observational data by partitioning data into subgroups, adjusting for confounders, and revealing hidden effect heterogeneity.

Contribution

It introduces a novel, interpretable recursive partitioning method that uncovers hidden subgroup effects and addresses Simpson's paradox under causal assumptions, improving upon existing approaches.

Findings

Builds simpler, interpretable trees

Effectively detects nested opposite effects

Ensures robust causal effect estimation

Abstract

Real-world observational datasets and machine learning have revolutionized data-driven decision-making, yet many models rely on empirical associations that may be misleading due to confounding and subgroup heterogeneity. Simpson's paradox exemplifies this challenge, where aggregated and subgroup-level associations contradict each other, leading to misleading conclusions. Existing methods provide limited support for detecting and interpreting such paradoxical associations, especially for practitioners without deep causal expertise. We introduce De-paradox Tree, an interpretable algorithm designed to uncover hidden subgroup patterns behind paradoxical associations under assumed causal structures involving confounders and effect heterogeneity. It employs novel split criteria and balancing-based procedures to adjust for confounders and homogenize heterogeneous effects through recursive partitioning. Compared to state-of-the-art methods, De-paradox Tree builds simpler, more interpretable trees, selects relevant covariates, and identifies nested opposite effects while ensuring robust estimation of causal effects when causally admissible variables are provided. Our approach addresses the limitations of traditional causal inference and machine learning methods by introducing an interpretable framework that supports non-expert practitioners while explicitly acknowledging causal assumptions and scope limitations, enabling more reliable and informed decision-making in complex observational data environments.