Guideline Learning for In-context Information Extraction

TL;DR

This paper introduces a Guideline Learning framework that enhances in-context information extraction by automatically generating and retrieving guidelines, significantly improving performance over traditional in-context learning methods.

Contribution

The paper proposes a novel Guideline Learning framework that automatically synthesizes and utilizes guidelines to improve in-context information extraction tasks.

Findings

GL significantly improves IE performance

Self-consistency active learning enhances guideline quality

Method outperforms baseline in event and relation extraction

Abstract

Large language models (LLMs) can perform a new task by merely conditioning on task instructions and a few input-output examples, without optimizing any parameters. This is called In-Context Learning (ICL). In-context Information Extraction (IE) has recently garnered attention in the research community. However, the performance of In-context IE generally lags behind the state-of-the-art supervised expert models. We highlight a key reason for this shortfall: underspecified task description. The limited-length context struggles to thoroughly express the intricate IE task instructions and various edge cases, leading to misalignment in task comprehension with humans. In this paper, we propose a Guideline Learning (GL) framework for In-context IE which reflectively learns and follows guidelines. During the learning phrase, GL automatically synthesizes a set of guidelines based on a few error cases, and during inference, GL retrieves helpful guidelines for better ICL. Moreover, we propose a self-consistency-based active learning method to enhance the efficiency of GL. Experiments on event extraction and relation extraction show that GL can significantly improve the performance of in-context IE.