Improving Low-Resource Morphological Inflection via Self-Supervised Objectives

TL;DR

This paper explores the use of self-supervised learning objectives to improve morphological inflection in extremely low-resource languages, finding that autoencoding and morpheme-aware masking enhance model performance.

Contribution

It introduces and evaluates various self-supervised auxiliary tasks for low-resource morphological inflection, highlighting the effectiveness of morpheme boundary sampling.

Findings

Autoencoding performs best with very limited data.

Character masked language modeling improves as data increases.

Morpheme boundary sampling consistently boosts performance.

Abstract

Self-supervised objectives have driven major advances in NLP by leveraging large-scale unlabeled data, but such resources are scarce for many of the world's languages. Surprisingly, they have not been explored much for character-level tasks, where smaller amounts of data have the potential to be beneficial. We investigate the effectiveness of self-supervised auxiliary tasks for morphological inflection -- a character-level task highly relevant for language documentation -- in extremely low-resource settings, training encoder-decoder transformers for 19 languages and 13 auxiliary objectives. Autoencoding yields the best performance when unlabeled data is very limited, while character masked language modeling (CMLM) becomes more effective as data availability increases. Though objectives with stronger inductive biases influence model predictions intuitively, they rarely outperform standard CMLM. However, sampling masks based on known morpheme boundaries consistently improves performance, highlighting a promising direction for low-resource morphological modeling.