Eeny, meeny, miny, moe. How to choose data for morphological inflection

TL;DR

This paper investigates active learning strategies for selecting data to improve morphological inflection in low-resource languages, demonstrating that confidence and entropy-based sampling enhance model performance, with oracle-based selection showing the most benefit.

Contribution

It compares four sampling strategies for morphological inflection across diverse languages, highlighting the effectiveness of confidence and entropy-based methods and analyzing their robustness.

Findings

Confidence and entropy-based sampling improve inflection accuracy.

Oracle-based selection yields the highest improvements.

Adding high-confidence or low-entropy forms can sometimes reduce performance.

Abstract

Data scarcity is a widespread problem in numerous natural language processing (NLP) tasks for low-resource languages. Within morphology, the labour-intensive work of tagging/glossing data is a serious bottleneck for both NLP and language documentation. Active learning (AL) aims to reduce the cost of data annotation by selecting data that is most informative for improving the model. In this paper, we explore four sampling strategies for the task of morphological inflection using a Transformer model: a pair of oracle experiments where data is chosen based on whether the model already can or cannot inflect the test forms correctly, as well as strategies based on high/low model confidence, entropy, as well as random selection. We investigate the robustness of each strategy across 30 typologically diverse languages. We also perform a more in-depth case study of Nat\"ugu. Our results show a clear benefit to selecting data based on model confidence and entropy. Unsurprisingly, the oracle experiment, where only incorrectly handled forms are chosen for further training, which is presented as a proxy for linguist/language consultant feedback, shows the most improvement. This is followed closely by choosing low-confidence and high-entropy predictions. We also show that despite the conventional wisdom of larger data sets yielding better accuracy, introducing more instances of high-confidence or low-entropy forms, or forms that the model can already inflect correctly, can reduce model performance.