Analysing Errors of Open Information Extraction Systems
Rudolf Schneider, Tom Oberhauser, Tobias Klatt, Felix A. Gers,, Alexander L\"oser
TL;DR
This paper benchmarks four popular Open Information Extraction systems across multiple datasets, analyzing their errors and performance to identify key research directions for future improvements.
Contribution
It introduces RelVis, a comprehensive benchmarking toolkit, and provides an in-depth error analysis of existing OIE systems on diverse datasets.
Findings
ClausIE and OpenIE 4.2 outperform others in certain metrics
Error analysis highlights common issues like relation extraction failures
Benchmarking reveals significant room for improvement in OIE accuracy
Abstract
We report results on benchmarking Open Information Extraction (OIE) systems using RelVis, a toolkit for benchmarking Open Information Extraction systems. Our comprehensive benchmark contains three data sets from the news domain and one data set from Wikipedia with overall 4522 labeled sentences and 11243 binary or n-ary OIE relations. In our analysis on these data sets we compared the performance of four popular OIE systems, ClausIE, OpenIE 4.2, Stanford OpenIE and PredPatt. In addition, we evaluated the impact of five common error classes on a subset of 749 n-ary tuples. From our deep analysis we unreveal important research directions for a next generation of OIE systems.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.