Inducing Causal World Models in LLMs for Zero-Shot Physical Reasoning

TL;DR

This paper presents CWMI, a framework that embeds causal physics into LLMs, enabling better zero-shot physical reasoning by learning cause-and-effect relationships through a dedicated module and intervention-based training.

Contribution

Introduces Causal World Model Induction (CWMI), integrating a causal physics module and intervention loss to enhance LLMs' understanding of physical dynamics.

Findings

CWMI outperforms existing LLMs on zero-shot physical reasoning benchmarks.

The model effectively learns cause-and-effect relationships from multimodal data.

Experimental results validate the importance of causal modeling for physical reasoning.

Abstract

Large Language Models (LLMs), despite their advanced linguistic capabilities, fundamentally lack an intuitive understanding of physical dynamics, which limits their effectiveness in real-world scenarios that require causal reasoning. In this paper, we introduce Causal World Model Induction (CWMI), a novel framework designed to embed an explicit model of causal physics within an LLM. Our approach incorporates a dedicated Causal Physics Module (CPM) and a new training objective called Causal Intervention Loss, encouraging the model to learn cause-and-effect relationships from multimodal data. By training the model to predict the outcomes of hypothetical interventions instead of merely capturing statistical correlations, CWMI develops a robust internal representation of physical laws. Experimental results show that CWMI significantly outperforms state-of-the-art LLMs on zero-shot physical reasoning tasks, including the PIQA benchmark and our newly proposed PhysiCa-Bench dataset. These findings demonstrate that inducing a causal world model is a critical step toward more reliable and generalizable AI systems.