Improving Open Information Extraction with Large Language Models: A Study on Demonstration Uncertainty

TL;DR

This paper explores how large language models can be improved for open information extraction by using in-context learning and uncertainty quantification, achieving competitive results with supervised methods.

Contribution

It introduces novel in-context learning strategies and a demonstration uncertainty quantification module to enhance LLM performance in OIE tasks.

Findings

LLMs can be improved with in-context learning for OIE.

Uncertainty quantification boosts confidence in generated relations.

Approach matches supervised methods on benchmark datasets.

Abstract

Open Information Extraction (OIE) task aims at extracting structured facts from unstructured text, typically in the form of (subject, relation, object) triples. Despite the potential of large language models (LLMs) like ChatGPT as a general task solver, they lag behind state-of-the-art (supervised) methods in OIE tasks due to two key issues. First, LLMs struggle to distinguish irrelevant context from relevant relations and generate structured output due to the restrictions on fine-tuning the model. Second, LLMs generates responses autoregressively based on probability, which makes the predicted relations lack confidence. In this paper, we assess the capabilities of LLMs in improving the OIE task. Particularly, we propose various in-context learning strategies to enhance LLM's instruction-following ability and a demonstration uncertainty quantification module to enhance the confidence of the generated relations. Our experiments on three OIE benchmark datasets show that our approach holds its own against established supervised methods, both quantitatively and qualitatively.