Instruct and Extract: Instruction Tuning for On-Demand Information Extraction

TL;DR

This paper introduces a new instruction-tuning paradigm for personalized, on-demand information extraction that allows users to specify extraction tasks and formats, supported by a new benchmark and a specialized model.

Contribution

It proposes the On-Demand Information Extraction paradigm, creates the InstructIE benchmark, and develops the ODIE model, advancing personalized extraction capabilities in NLP.

Findings

ODIE outperforms existing open-source models of similar size.

The benchmark facilitates research in personalized information extraction.

The approach enables flexible, user-specified extraction tasks.

Abstract

Large language models with instruction-following capabilities open the door to a wider group of users. However, when it comes to information extraction - a classic task in natural language processing - most task-specific systems cannot align well with long-tail ad hoc extraction use cases for non-expert users. To address this, we propose a novel paradigm, termed On-Demand Information Extraction, to fulfill the personalized demands of real-world users. Our task aims to follow the instructions to extract the desired content from the associated text and present it in a structured tabular format. The table headers can either be user-specified or inferred contextually by the model. To facilitate research in this emerging area, we present a benchmark named InstructIE, inclusive of both automatically generated training data, as well as the human-annotated test set. Building on InstructIE, we further develop an On-Demand Information Extractor, ODIE. Comprehensive evaluations on our benchmark reveal that ODIE substantially outperforms the existing open-source models of similar size. Our code and dataset are released on https://github.com/yzjiao/On-Demand-IE.