$\textit{BenchIE}^{FL}$ : A Manually Re-Annotated Fact-Based Open Information Extraction Benchmark

TL;DR

BenchIE^{FL} is a manually re-annotated benchmark for open information extraction that improves upon previous benchmarks by reducing errors and omissions, enabling more accurate evaluation of OIE systems.

Contribution

It introduces a refined, error-reduced version of the BenchIE benchmark, enhancing the reliability of OIE system evaluations.

Findings

Fewer errors and omissions in candidate fact matching.

More accurate assessment of OIE system performance.

Supports insightful analysis of OIE extractors.

Abstract

Open Information Extraction (OIE) is a field of natural language processing that aims to present textual information in a format that allows it to be organized, analyzed and reflected upon. Numerous OIE systems are developed, claiming ever-increasing performance, marking the need for objective benchmarks. BenchIE is the latest reference we know of. Despite being very well thought out, we noticed a number of issues we believe are limiting. Therefore, we propose $\textit{BenchIE}^{FL}$, a new OIE benchmark which fully enforces the principles of BenchIE while containing fewer errors, omissions and shortcomings when candidate facts are matched towards reference ones. $\textit{BenchIE}^{FL}$ allows insightful conclusions to be drawn on the actual performance of OIE extractors.