Instruction-Tuning LLMs for Event Extraction with Annotation Guidelines

TL;DR

This paper investigates how annotation guidelines influence instruction-tuning of large language models for event extraction, showing their benefits in data-rich and low-data scenarios, especially for generalization and rare event types.

Contribution

It introduces the use of annotation guidelines in instruction-tuning for event extraction and evaluates their impact across different data settings and sources.

Findings

Guidelines improve performance with sufficient data.

Guidelines enhance cross-schema generalization.

Guidelines help detect low-frequency event types.

Abstract

In this work, we study the effect of annotation guidelines -- textual descriptions of event types and arguments, when instruction-tuning large language models for event extraction. We conducted a series of experiments with both human-provided and machine-generated guidelines in both full- and low-data settings. Our results demonstrate the promise of annotation guidelines when there is a decent amount of training data and highlight its effectiveness in improving cross-schema generalization and low-frequency event-type performance.