Finding Non-Redundant Simpson's Paradox from Multidimensional Data

TL;DR

This paper introduces a novel framework for efficiently discovering non-redundant instances of Simpson's paradox in large multidimensional datasets, addressing redundancy issues that hinder previous methods.

Contribution

It formalizes types of redundancy in Simpson's paradox, proposes a concise representation framework, and develops algorithms that significantly improve detection efficiency and interpretability.

Findings

Redundant paradoxes can constitute over 40% of all detected paradoxes.

Algorithms reduce runtime by up to 60% on large datasets.

The framework scales to millions of records and maintains robustness under data perturbation.

Abstract

Simpson's paradox, a long-standing statistical phenomenon, describes the reversal of an observed association when data are disaggregated into sub-populations. It has critical implications across statistics, epidemiology, economics, and causal inference. Existing methods for detecting Simpson's paradox overlook a key issue: many paradoxes are redundant, arising from equivalent selections of data subsets, identical partitioning of sub-populations, and correlated outcome variables, which obscure essential patterns and inflate computational cost. In this paper, we present the first framework for discovering non-redundant Simpson's paradoxes. We formalize three types of redundancy - sibling child, separator, and statistic equivalence - and show that redundancy forms an equivalence relation. Leveraging this insight, we propose a concise representation framework for systematically organizing redundant paradoxes and design efficient algorithms that integrate depth-first materialization of the base table with redundancy-aware paradox discovery. Experiments on real-world datasets and synthetic benchmarks show that redundant paradoxes are widespread, on some real datasets constituting over 40% of all paradoxes, while our algorithms scale to millions of records, reduce run time by up to 60%, and discover paradoxes that are structurally robust under data perturbation. These results demonstrate that Simpson's paradoxes can be efficiently identified, concisely summarized, and meaningfully interpreted in large multidimensional datasets.