Think before You Simulate: Symbolic Reasoning to Orchestrate Neural Computation for Counterfactual Question Answering

TL;DR

This paper presents a neuro-symbolic approach using causal graphs and Answer Set Programming to improve counterfactual reasoning in video understanding, achieving state-of-the-art results on benchmarks like CLEVRER.

Contribution

It introduces a novel method combining symbolic causal reasoning with neural perception modules, enhancing counterfactual question answering capabilities.

Findings

Achieves state-of-the-art performance on CLEVRER benchmark.

Leverages large language models as proxies for dynamics simulators.

Improves counterfactual reasoning accuracy through symbolic causal prompts.

Abstract

Causal and temporal reasoning about video dynamics is a challenging problem. While neuro-symbolic models that combine symbolic reasoning with neural-based perception and prediction have shown promise, they exhibit limitations, especially in answering counterfactual questions. This paper introduces a method to enhance a neuro-symbolic model for counterfactual reasoning, leveraging symbolic reasoning about causal relations among events. We define the notion of a causal graph to represent such relations and use Answer Set Programming (ASP), a declarative logic programming method, to find how to coordinate perception and simulation modules. We validate the effectiveness of our approach on two benchmarks, CLEVRER and CRAFT. Our enhancement achieves state-of-the-art performance on the CLEVRER challenge, significantly outperforming existing models. In the case of the CRAFT benchmark, we leverage a large pre-trained language model, such as GPT-3.5 and GPT-4, as a proxy for a dynamics simulator. Our findings show that this method can further improve its performance on counterfactual questions by providing alternative prompts instructed by symbolic causal reasoning.