InstructUIE: Multi-task Instruction Tuning for Unified Information Extraction

TL;DR

InstructUIE is a unified instruction-tuned framework for information extraction that models multiple tasks simultaneously, outperforming existing large models especially in zero-shot scenarios, validated on a new diverse benchmark.

Contribution

The paper introduces InstructUIE, a novel instruction tuning approach for multi-task information extraction, and presents IE INSTRUCTIONS, a comprehensive benchmark for evaluation.

Findings

InstructUIE achieves comparable results to BERT in supervised settings.

It significantly outperforms GPT-3.5 in zero-shot settings.

The benchmark includes 32 diverse datasets in a unified format.

Abstract

Large language models have unlocked strong multi-task capabilities from reading instructive prompts. However, recent studies have shown that existing large models still have difficulty with information extraction tasks. For example, gpt-3.5-turbo achieved an F1 score of 18.22 on the Ontonotes dataset, which is significantly lower than the state-of-the-art performance. In this paper, we propose InstructUIE, a unified information extraction framework based on instruction tuning, which can uniformly model various information extraction tasks and capture the inter-task dependency. To validate the proposed method, we introduce IE INSTRUCTIONS, a benchmark of 32 diverse information extraction datasets in a unified text-to-text format with expert-written instructions. Experimental results demonstrate that our method achieves comparable performance to Bert in supervised settings and significantly outperforms the state-of-the-art and gpt3.5 in zero-shot settings.