ExpliCa: Evaluating Explicit Causal Reasoning in Large Language Models

TL;DR

ExpliCa introduces a new dataset to evaluate large language models' ability to perform explicit causal reasoning, revealing current models' limitations and their confusion between causal and temporal relations.

Contribution

The paper presents ExpliCa, a novel dataset that assesses LLMs on explicit causal reasoning with diverse linguistic structures and human ratings, highlighting model challenges.

Findings

Models struggle to reach 80% accuracy on causal reasoning tasks.

Models often confuse temporal relations with causal ones.

Model performance varies with linguistic order and size.

Abstract

Large Language Models (LLMs) are increasingly used in tasks requiring interpretive and inferential accuracy. In this paper, we introduce ExpliCa, a new dataset for evaluating LLMs in explicit causal reasoning. ExpliCa uniquely integrates both causal and temporal relations presented in different linguistic orders and explicitly expressed by linguistic connectives. The dataset is enriched with crowdsourced human acceptability ratings. We tested LLMs on ExpliCa through prompting and perplexity-based metrics. We assessed seven commercial and open-source LLMs, revealing that even top models struggle to reach 0.80 accuracy. Interestingly, models tend to confound temporal relations with causal ones, and their performance is also strongly influenced by the linguistic order of the events. Finally, perplexity-based scores and prompting performance are differently affected by model size.