Reflect then Learn: Active Prompting for Information Extraction Guided by Introspective Confusion

TL;DR

This paper introduces Active Prompting for Information Extraction (APIE), a method where LLMs assess their own confusion to select challenging samples, improving extraction accuracy and robustness.

Contribution

The paper proposes a novel active prompting framework guided by dual-component uncertainty, addressing model fallibility in few-shot IE tasks.

Findings

APIE outperforms strong baselines on four benchmarks.

It improves extraction accuracy and robustness.

Dual-level uncertainty effectively guides sample selection.

Abstract

Large Language Models (LLMs) show remarkable potential for few-shot information extraction (IE), yet their performance is highly sensitive to the choice of in-context examples. Conventional selection strategies often fail to provide informative guidance, as they overlook a key source of model fallibility: confusion stemming not just from semantic content, but also from the generation of well-structured formats required by IE tasks. To address this, we introduce Active Prompting for Information Extraction (APIE), a novel active prompting framework guided by a principle we term introspective confusion. Our method empowers an LLM to assess its own confusion through a dual-component uncertainty metric that uniquely quantifies both Format Uncertainty (difficulty in generating correct syntax) and Content Uncertainty (inconsistency in extracted semantics). By ranking unlabeled data with this comprehensive score, our framework actively selects the most challenging and informative samples to serve as few-shot exemplars. Extensive experiments on four benchmarks show that our approach consistently outperforms strong baselines, yielding significant improvements in both extraction accuracy and robustness. Our work highlights the critical importance of a fine-grained, dual-level view of model uncertainty when it comes to building effective and reliable structured generation systems.