Investigating Multi-source Active Learning for Natural Language Inference

TL;DR

This paper examines the challenges of multi-source active learning in NLP, revealing that common strategies struggle with outliers and source variability, but can improve with outlier removal and difficulty-aware testing.

Contribution

It demonstrates the limitations of existing active learning methods in multi-source settings and proposes analysis techniques to understand and improve their performance.

Findings

Uncertainty-based strategies perform poorly with outliers in multi-source data.

Removing outliers allows strategies to outperform random selection.

Different sources produce varying types of outliers and learnability challenges.

Abstract

In recent years, active learning has been successfully applied to an array of NLP tasks. However, prior work often assumes that training and test data are drawn from the same distribution. This is problematic, as in real-life settings data may stem from several sources of varying relevance and quality. We show that four popular active learning schemes fail to outperform random selection when applied to unlabelled pools comprised of multiple data sources on the task of natural language inference. We reveal that uncertainty-based strategies perform poorly due to the acquisition of collective outliers, i.e., hard-to-learn instances that hamper learning and generalization. When outliers are removed, strategies are found to recover and outperform random baselines. In further analysis, we find that collective outliers vary in form between sources, and show that hard-to-learn data is not always categorically harmful. Lastly, we leverage dataset cartography to introduce difficulty-stratified testing and find that different strategies are affected differently by example learnability and difficulty.