CausalMan: A physics-based simulator for large-scale causality

TL;DR

CausalMan is a physics-based large-scale causal simulator modeled after a real-world production line, designed to benchmark causal inference methods with diverse mechanisms and behaviors.

Contribution

We introduce CausalMan, a novel, realistic causal simulator with diverse mechanisms, and provide datasets and evaluations to facilitate benchmarking of causal inference approaches.

Findings

Many state-of-the-art methods are inadequate on CausalMan.

Performance varies significantly across methods in runtime and memory.

CausalMan enables realistic benchmarking of causal models.

Abstract

A comprehensive understanding of causality is critical for navigating and operating within today's complex real-world systems. The absence of realistic causal models with known data generating processes complicates fair benchmarking. In this paper, we present the CausalMan simulator, modeled after a real-world production line. The simulator features a diverse range of linear and non-linear mechanisms and challenging-to-predict behaviors, such as discrete mode changes. We demonstrate the inadequacy of many state-of-the-art approaches and analyze the significant differences in their performance and tractability, both in terms of runtime and memory complexity. As a contribution, we will release the CausalMan large-scale simulator. We present two derived datasets, and perform an extensive evaluation of both.