Scoping Review of Active Learning Strategies and their Evaluation Environments for Entity Recognition Tasks

TL;DR

This scoping review comprehensively analyzes active learning strategies for entity recognition in NLP, highlighting their evaluation environments, datasets, and open research questions to guide future empirical comparisons.

Contribution

It categorizes 106 active learning strategies, assesses their evaluation environments, and identifies gaps in hardware and dataset accessibility for NLP entity recognition.

Findings

All studies used F1-score as an evaluation metric.

Most datasets contained newspaper or biomedical data.

Significant open questions remain in strategy selection and evaluation.

Abstract

We conducted a scoping review for active learning in the domain of natural language processing (NLP), which we summarize in accordance with the PRISMA-ScR guidelines as follows: Objective: Identify active learning strategies that were proposed for entity recognition and their evaluation environments (datasets, metrics, hardware, execution time). Design: We used Scopus and ACM as our search engines. We compared the results with two literature surveys to assess the search quality. We included peer-reviewed English publications introducing or comparing active learning strategies for entity recognition. Results: We analyzed 62 relevant papers and identified 106 active learning strategies. We grouped them into three categories: exploitation-based (60x), exploration-based (14x), and hybrid strategies (32x). We found that all studies used the F1-score as an evaluation metric. Information about hardware (6x) and execution time (13x) was only occasionally included. The 62 papers used 57 different datasets to evaluate their respective strategies. Most datasets contained newspaper articles or biomedical/medical data. Our analysis revealed that 26 out of 57 datasets are publicly accessible. Conclusion: Numerous active learning strategies have been identified, along with significant open questions that still need to be addressed. Researchers and practitioners face difficulties when making data-driven decisions about which active learning strategy to adopt. Conducting comprehensive empirical comparisons using the evaluation environment proposed in this study could help establish best practices in the domain.