SERE: Structural Example Retrieval for Enhancing LLMs in Event Causality Identification

TL;DR

SERE is a framework that enhances Large Language Models' ability to identify event causality by retrieving structurally relevant examples using conceptual, syntactic, and causal pattern metrics.

Contribution

It introduces a novel structural example retrieval method that improves LLMs' causal reasoning in event causality identification tasks.

Findings

SERE significantly improves ECI accuracy across multiple datasets.

Structural retrieval reduces causal hallucination in LLMs.

The source code is publicly available at https://github.com/DMIRLAB-Group/SERE.

Abstract

Event Causality Identification (ECI) requires models to determine whether a given pair of events in a context exhibits a causal relationship. While Large Language Models (LLMs) have demonstrated strong performance across various NLP tasks, their effectiveness in ECI remains limited due to biases in causal reasoning, often leading to overprediction of causal relationships (causal hallucination). To mitigate these issues and enhance LLM performance in ECI, we propose SERE, a structural example retrieval framework that leverages LLMs' few-shot learning capabilities. SERE introduces an innovative retrieval mechanism based on three structural concepts: (i) Conceptual Path Metric, which measures the conceptual relationship between events using edit distance in ConceptNet; (ii) Syntactic Metric, which quantifies structural similarity through tree edit distance on syntactic trees; and (iii) Causal Pattern Filtering, which filters examples based on predefined causal structures using LLMs. By integrating these structural retrieval strategies, SERE selects more relevant examples to guide LLMs in causal reasoning, mitigating bias and improving accuracy in ECI tasks. Extensive experiments on multiple ECI datasets validate the effectiveness of SERE. The source code is publicly available at https://github.com/DMIRLAB-Group/SERE.