BENGAL: An Automatic Benchmark Generator for Entity Recognition and Linking

TL;DR

BENGAL is an automatic, cost-effective benchmark generator for entity recognition and linking that produces error-free gold standards, facilitating rapid and reliable evaluation of NLP tools across multiple languages.

Contribution

This paper introduces BENGAL, a novel method for automatically generating gold standard benchmarks for entity recognition and linking, reducing manual effort and errors.

Findings

BENGAL-generated benchmarks are comparable to manually created ones in quality.

The approach is easily adaptable to multiple languages, demonstrated with Portuguese and Spanish.

Automatic benchmarks can effectively evaluate NLP tools, showing consistent results.

Abstract

The manual creation of gold standards for named entity recognition and entity linking is time- and resource-intensive. Moreover, recent works show that such gold standards contain a large proportion of mistakes in addition to being difficult to maintain. We hence present BENGAL, a novel automatic generation of such gold standards as a complement to manually created benchmarks. The main advantage of our benchmarks is that they can be readily generated at any time. They are also cost-effective while being guaranteed to be free of annotation errors. We compare the performance of 11 tools on benchmarks in English generated by BENGAL and on 16benchmarks created manually. We show that our approach can be ported easily across languages by presenting results achieved by 4 tools on both Brazilian Portuguese and Spanish. Overall, our results suggest that our automatic benchmark generation approach can create varied benchmarks that have characteristics similar to those of existing benchmarks. Our approach is open-source. Our experimental results are available at http://faturl.com/bengalexpinlg and the code at https://github.com/dice-group/BENGAL.