WikiCausal: Corpus and Evaluation Framework for Causal Knowledge Graph Construction

TL;DR

This paper introduces a new corpus, task, and evaluation framework for constructing causal knowledge graphs from Wikipedia, enabling more effective assessment and development of automated causal relation extraction methods.

Contribution

It provides a comprehensive dataset and evaluation framework for causal knowledge graph construction, addressing limitations of previous low-level or small-scale evaluations.

Findings

Effective use of Large Language Models for evaluation

Neural models improve causal relation extraction

Framework facilitates model comparison and selection

Abstract

Recently, there has been an increasing interest in the construction of general-domain and domain-specific causal knowledge graphs. Such knowledge graphs enable reasoning for causal analysis and event prediction, and so have a range of applications across different domains. While great progress has been made toward automated construction of causal knowledge graphs, the evaluation of such solutions has either focused on low-level tasks (e.g., cause-effect phrase extraction) or on ad hoc evaluation data and small manual evaluations. In this paper, we present a corpus, task, and evaluation framework for causal knowledge graph construction. Our corpus consists of Wikipedia articles for a collection of event-related concepts in Wikidata. The task is to extract causal relations between event concepts from the corpus. The evaluation is performed in part using existing causal relations in Wikidata to measure recall, and in part using Large Language Models to avoid the need for manual or crowd-sourced evaluation. We evaluate a pipeline for causal knowledge graph construction that relies on neural models for question answering and concept linking, and show how the corpus and the evaluation framework allow us to effectively find the right model for each task. The corpus and the evaluation framework are publicly available.