Exploring the Feasibility of ChatGPT for Event Extraction

TL;DR

This paper investigates the potential of ChatGPT for event extraction tasks, highlighting its limitations in performance, robustness, and sensitivity to prompts compared to specialized models.

Contribution

It provides an empirical evaluation of ChatGPT's capabilities for event extraction, revealing significant challenges and the need for further prompt engineering.

Findings

ChatGPT achieves about 51% of the performance of task-specific models.

It is not robust and performance does not improve with prompt refinement.

Highly sensitive to different prompt styles.

Abstract

Event extraction is a fundamental task in natural language processing that involves identifying and extracting information about events mentioned in text. However, it is a challenging task due to the lack of annotated data, which is expensive and time-consuming to obtain. The emergence of large language models (LLMs) such as ChatGPT provides an opportunity to solve language tasks with simple prompts without the need for task-specific datasets and fine-tuning. While ChatGPT has demonstrated impressive results in tasks like machine translation, text summarization, and question answering, it presents challenges when used for complex tasks like event extraction. Unlike other tasks, event extraction requires the model to be provided with a complex set of instructions defining all event types and their schemas. To explore the feasibility of ChatGPT for event extraction and the challenges it poses, we conducted a series of experiments. Our results show that ChatGPT has, on average, only 51.04% of the performance of a task-specific model such as EEQA in long-tail and complex scenarios. Our usability testing experiments indicate that ChatGPT is not robust enough, and continuous refinement of the prompt does not lead to stable performance improvements, which can result in a poor user experience. Besides, ChatGPT is highly sensitive to different prompt styles.