The Practical Challenges of Active Learning: Lessons Learned from Live Experimentation

TL;DR

This paper examines the practical challenges of implementing active learning in real-world scenarios through a live experiment on Thai text sentence annotation, highlighting environmental impacts and operational issues.

Contribution

It provides empirical insights into the difficulties faced when deploying active learning in live settings and discusses strategies to address these challenges.

Findings

Active learning's effectiveness is influenced by environmental changes.

Practical challenges include data distribution shifts and operational constraints.

Live experimentation reveals issues not apparent in controlled studies.

Abstract

We tested in a live setting the use of active learning for selecting text sentences for human annotations used in training a Thai segmentation machine learning model. In our study, two concurrent annotated samples were constructed, one through random sampling of sentences from a text corpus, and the other through model-based scoring and ranking of sentences from the same corpus. In the course of the experiment, we observed the effect of significant changes to the learning environment which are likely to occur in real-world learning tasks. We describe how our active learning strategy interacted with these events and discuss other practical challenges encountered in using active learning in the live setting.