Large Language Models for Causal Relations Extraction in Social Media: A Validation Framework for Disaster Intelligence

TL;DR

This paper evaluates the effectiveness of Large Language Models in extracting causal relations from disaster-related social media posts, proposing an evaluation framework and analyzing potential and risks.

Contribution

It introduces an expert-grounded evaluation framework for LLMs in causal relation extraction and assesses their reliability in disaster contexts.

Findings

LLMs can generate causal graphs that partially align with reference data.

There are significant risks of LLMs reflecting prior knowledge rather than actual post-event evidence.

The framework helps identify when LLMs provide trustworthy causal information.

Abstract

During disasters, extracting causal relations from social media can strengthen situational awareness by identifying factors linked to casualties, physical damage, infrastructure disruption, and cascading impacts. However, disaster-related posts are often informal, fragmented, and context-dependent, and they may describe personal experiences rather than explicit causal relations. In this work, we examine whether Large Language Models (LLMs) can effectively extract causal relations from disaster-related social media posts. To this end, we (1) propose an expert-grounded evaluation framework that compares LLM-generated causal graphs with reference graphs derived from disaster-specific reports and (2) assess whether the extracted relations are supported by post-event evidence or instead reflect model priors. Our findings highlight both the potential and risks of using LLMs for causal relation extraction in disaster decision-support systems.