Mastering the Task of Open Information Extraction with Large Language Models and Consistent Reasoning Environment

TL;DR

This paper demonstrates that large language models, when provided with a carefully constructed reasoning environment and positive demonstrations, can effectively perform open information extraction, surpassing existing supervised methods on standard benchmarks.

Contribution

The authors introduce a novel method to estimate syntactic distribution discrepancies and establish a reasoning environment for LLMs, significantly improving OIE performance.

Findings

Achieved 55.3 F1 score on CaRB benchmark, outperforming state-of-the-art supervised methods.

Demonstrated generalization to other information extraction tasks with notable improvements.

Proved effectiveness of reasoning environment and positive demonstrations in enhancing LLM capabilities.

Abstract

Open Information Extraction (OIE) aims to extract objective structured knowledge from natural texts, which has attracted growing attention to build dedicated models with human experience. As the large language models (LLMs) have exhibited remarkable in-context learning capabilities, a question arises as to whether the task of OIE can be effectively tackled with this paradigm? In this paper, we explore solving the OIE problem by constructing an appropriate reasoning environment for LLMs. Specifically, we first propose a method to effectively estimate the discrepancy of syntactic distribution between a LLM and test samples, which can serve as correlation evidence for preparing positive demonstrations. Upon the evidence, we introduce a simple yet effective mechanism to establish the reasoning environment for LLMs on specific tasks. Without bells and whistles, experimental results on the standard CaRB benchmark demonstrate that our $6$-shot approach outperforms state-of-the-art supervised method, achieving an $55.3$ $F_1$ score. Further experiments on TACRED and ACE05 show that our method can naturally generalize to other information extraction tasks, resulting in improvements of $5.7$ and $6.8$ $F_1$ scores, respectively.