Using Error Decay Prediction to Overcome Practical Issues of Deep Active Learning for Named Entity Recognition

TL;DR

This paper introduces a transparent batch active sampling framework for deep active learning in NER, addressing practical issues like black-box model use, noise robustness, and transparency, and demonstrating improved performance and insights.

Contribution

It proposes a novel error decay prediction-based sampling method that enhances robustness and transparency in deep active learning for NER tasks.

Findings

Outperforms diversification-based methods for black-box NER models

Increases robustness to labeling noise when combined with uncertainty sampling

Provides insights into when different active sampling strategies are effective

Abstract

Existing deep active learning algorithms achieve impressive sampling efficiency on natural language processing tasks. However, they exhibit several weaknesses in practice, including (a) inability to use uncertainty sampling with black-box models, (b) lack of robustness to labeling noise, and (c) lack of transparency. In response, we propose a transparent batch active sampling framework by estimating the error decay curves of multiple feature-defined subsets of the data. Experiments on four named entity recognition (NER) tasks demonstrate that the proposed methods significantly outperform diversification-based methods for black-box NER taggers, and can make the sampling process more robust to labeling noise when combined with uncertainty-based methods. Furthermore, the analysis of experimental results sheds light on the weaknesses of different active sampling strategies, and when traditional uncertainty-based or diversification-based methods can be expected to work well.