AnnIE: An Annotation Platform for Constructing Complete Open Information Extraction Benchmark

TL;DR

AnnIE is an interactive platform designed to create comprehensive OIE benchmarks by manually annotating complete facts, revealing that current benchmarks may overestimate system performance and highlighting the need for more robust evaluation.

Contribution

AnnIE introduces a modular annotation platform for constructing complete, fact-oriented OIE benchmarks, enabling more realistic performance evaluation of OIE systems.

Findings

Existing benchmarks are overly lenient.

OIE systems are less robust than previously reported.

AnnIE facilitates the creation of diverse, complete fact benchmarks.

Abstract

Open Information Extraction (OIE) is the task of extracting facts from sentences in the form of relations and their corresponding arguments in schema-free manner. Intrinsic performance of OIE systems is difficult to measure due to the incompleteness of existing OIE benchmarks: the ground truth extractions do not group all acceptable surface realizations of the same fact that can be extracted from a sentence. To measure performance of OIE systems more realistically, it is necessary to manually annotate complete facts (i.e., clusters of all acceptable surface realizations of the same fact) from input sentences. We propose AnnIE: an interactive annotation platform that facilitates such challenging annotation tasks and supports creation of complete fact-oriented OIE evaluation benchmarks. AnnIE is modular and flexible in order to support different use case scenarios (i.e., benchmarks covering different types of facts). We use AnnIE to build two complete OIE benchmarks: one with verb-mediated facts and another with facts encompassing named entities. Finally, we evaluate several OIE systems on our complete benchmarks created with AnnIE. Our results suggest that existing incomplete benchmarks are overly lenient, and that OIE systems are not as robust as previously reported. We publicly release AnnIE under non-restrictive license.