Towards Generalized Open Information Extraction

TL;DR

This paper introduces GLOBE, a multi-domain OpenIE benchmark, and DragonIE, a minimalist graph-based model, to improve the generalization of OpenIE systems across unseen domains, addressing a key challenge in the field.

Contribution

The paper presents GLOBE for evaluating domain robustness and DragonIE, a novel graph-based model, advancing OpenIE's ability to generalize across diverse data distributions.

Findings

DragonIE outperforms previous models in both in-domain and out-of-domain tests.

Significant performance degradation occurs when models face domain shifts, highlighting the challenge.

DragonIE achieves up to 6.0% higher F1 score compared to prior methods.

Abstract

Open Information Extraction (OpenIE) facilitates the open-domain discovery of textual facts. However, the prevailing solutions evaluate OpenIE models on in-domain test sets aside from the training corpus, which certainly violates the initial task principle of domain-independence. In this paper, we propose to advance OpenIE towards a more realistic scenario: generalizing over unseen target domains with different data distributions from the source training domains, termed Generalized OpenIE. For this purpose, we first introduce GLOBE, a large-scale human-annotated multi-domain OpenIE benchmark, to examine the robustness of recent OpenIE models to domain shifts, and the relative performance degradation of up to 70% implies the challenges of generalized OpenIE. Then, we propose DragonIE, which explores a minimalist graph expression of textual fact: directed acyclic graph, to improve the OpenIE generalization. Extensive experiments demonstrate that DragonIE beats the previous methods in both in-domain and out-of-domain settings by as much as 6.0% in F1 score absolutely, but there is still ample room for improvement.