Active Learning for Sequence Tagging with Deep Pre-trained Models and Bayesian Uncertainty Estimates

TL;DR

This paper explores combining active learning with deep pre-trained models and Bayesian uncertainty estimates to efficiently reduce annotation efforts in sequence tagging tasks, including practical model distillation for better performance.

Contribution

It provides the first comprehensive empirical analysis of Bayesian uncertainty methods with deep pre-trained models in active learning for sequence tagging, and demonstrates the effectiveness of distilled models.

Findings

Bayesian methods improve active learning efficiency

Distilled Transformer models outperform full-size models in active learning

Optimal uncertainty estimation techniques vary by model type

Abstract

Annotating training data for sequence tagging of texts is usually very time-consuming. Recent advances in transfer learning for natural language processing in conjunction with active learning open the possibility to significantly reduce the necessary annotation budget. We are the first to thoroughly investigate this powerful combination for the sequence tagging task. We conduct an extensive empirical study of various Bayesian uncertainty estimation methods and Monte Carlo dropout options for deep pre-trained models in the active learning framework and find the best combinations for different types of models. Besides, we also demonstrate that to acquire instances during active learning, a full-size Transformer can be substituted with a distilled version, which yields better computational performance and reduces obstacles for applying deep active learning in practice.