LLMs Learn Task Heuristics from Demonstrations: A Heuristic-Driven Prompting Strategy for Document-Level Event Argument Extraction

TL;DR

This paper presents a heuristic-driven prompting strategy for document-level event argument extraction that enables LLMs to learn task-specific heuristics from demonstrations, improving performance with fewer labeled examples.

Contribution

The paper introduces HD-LoA prompting, a novel heuristic-driven demonstration construction and analogical reasoning approach, enhancing LLMs' ability to perform document-level EAE and other NLP tasks.

Findings

Outperforms existing prompting and few-shot learning methods on EAE datasets.

Effective in diverse tasks like sentiment analysis and natural language inference.

Enables LLMs to learn task heuristics from demonstrations.

Abstract

In this study, we investigate in-context learning (ICL) in document-level event argument extraction (EAE) to alleviate the dependency on large-scale labeled data for this task. We introduce the Heuristic-Driven Link-of-Analogy (HD-LoA) prompting to address the challenge of example selection and to develop a prompting strategy tailored for EAE. Specifically, we hypothesize and validate that LLMs learn task-specific heuristics from demonstrations via ICL. Building upon this hypothesis, we introduce an explicit heuristic-driven demonstration construction approach, which transforms the haphazard example selection process into a methodical method that emphasizes task heuristics. Additionally, inspired by the analogical reasoning of human, we propose the link-of-analogy prompting, which enables LLMs to process new situations by drawing analogies to known situations, enhancing their performance on unseen classes beyond limited ICL examples. Experiments show that our method outperforms existing prompting methods and few-shot supervised learning methods on document-level EAE datasets. Additionally, the HD-LoA prompting shows effectiveness in diverse tasks like sentiment analysis and natural language inference, demonstrating its broad adaptability.