Benchmarking Large Language Models with Augmented Instructions for Fine-grained Information Extraction

TL;DR

This paper presents a new benchmark dataset and evaluation framework for fine-grained information extraction using large language models, emphasizing augmented instructions and analyzing model performance and generalization capabilities.

Contribution

It introduces a fine-grained IE benchmark with augmented instructions for LLMs and evaluates different models' abilities to generalize to unseen information types.

Findings

Encoder-decoder models like T5 perform well on unseen types.

ChatGPT shows high adaptability to new task forms.

Model performance depends on architecture, data diversity, and learning techniques.

Abstract

Information Extraction (IE) is an essential task in Natural Language Processing. Traditional methods have relied on coarse-grained extraction with simple instructions. However, with the emergence of Large Language Models (LLMs), there is a need to adapt IE techniques to leverage the capabilities of these models. This paper introduces a fine-grained IE benchmark dataset tailored for LLMs, employing augmented instructions for each information type, which includes task descriptions, extraction rules, output formats, and examples. Through extensive evaluations, we observe that encoder-decoder models, particularly T5 and FLAN-T5, perform well in generalizing to unseen information types, while ChatGPT exhibits greater adaptability to new task forms. Our results also indicate that performance is not solely dictated by model scale, and highlight the significance of architecture, data diversity, and learning techniques. This work paves the way for a more refined and versatile utilization of LLMs in Information Extraction.