Com$^2$: A Causal-Guided Benchmark for Exploring Complex Commonsense Reasoning in Large Language Models

TL;DR

This paper introduces Com$^2$, a benchmark for evaluating large language models on complex, implicit commonsense reasoning using causal event graphs and slow thinking, revealing current limitations and potential improvements.

Contribution

It presents a novel benchmark incorporating causal graphs and theory-based modifications to assess and enhance LLMs' complex commonsense reasoning capabilities.

Findings

LLMs struggle with reasoning depth and breadth in complex commonsense tasks.

Post-training and slow thinking improve LLM performance on the benchmark.

The benchmark provides a structured way to evaluate complex commonsense reasoning in LLMs.

Abstract

Large language models (LLMs) have mastered abundant simple and explicit commonsense knowledge through pre-training, enabling them to achieve human-like performance in simple commonsense reasoning. Nevertheless, LLMs struggle to reason with complex and implicit commonsense knowledge that is derived from simple ones (such as understanding the long-term effects of certain events), an aspect humans tend to focus on more. Existing works focus on complex tasks like math and code, while complex commonsense reasoning remains underexplored due to its uncertainty and lack of structure. To fill this gap and align with real-world concerns, we propose a benchmark Com$^2$ focusing on complex commonsense reasoning. We first incorporate causal event graphs to serve as structured complex commonsense. Then we adopt causal theory~(e.g., intervention) to modify the causal event graphs and obtain different scenarios that meet human concerns. Finally, an LLM is employed to synthesize examples with slow thinking, which is guided by the logical relationships in the modified causal graphs. Furthermore, we use detective stories to construct a more challenging subset. Experiments show that LLMs struggle in reasoning depth and breadth, while post-training and slow thinking can alleviate this. The code and data are available at https://github.com/Waste-Wood/Com2.