Are LLMs Good Annotators for Discourse-level Event Relation Extraction?

TL;DR

This paper evaluates the effectiveness of large language models like GPT-3.5 and LLaMA-2 in discourse-level event relation extraction, revealing significant limitations compared to supervised models, despite some improvements with fine-tuning.

Contribution

It provides a comprehensive assessment of LLMs for discourse-level event relation extraction, highlighting their weaknesses and the challenges in scaling fine-tuning methods.

Findings

LLMs underperform compared to supervised baselines.

Fine-tuning improves LLM performance but does not scale well.

LLMs tend to fabricate event mentions and struggle with complex relations.

Abstract

Large Language Models (LLMs) have demonstrated proficiency in a wide array of natural language processing tasks. However, its effectiveness over discourse-level event relation extraction (ERE) tasks remains unexplored. In this paper, we assess the effectiveness of LLMs in addressing discourse-level ERE tasks characterized by lengthy documents and intricate relations encompassing coreference, temporal, causal, and subevent types. Evaluation is conducted using an commercial model, GPT-3.5, and an open-source model, LLaMA-2. Our study reveals a notable underperformance of LLMs compared to the baseline established through supervised learning. Although Supervised Fine-Tuning (SFT) can improve LLMs performance, it does not scale well compared to the smaller supervised baseline model. Our quantitative and qualitative analysis shows that LLMs have several weaknesses when applied for extracting event relations, including a tendency to fabricate event mentions, and failures to capture transitivity rules among relations, detect long distance relations, or comprehend contexts with dense event mentions. Code available at: https://github.com/WeiKangda/LLM-ERE.git.