Multilingual unsupervised sequence segmentation transfers to extremely low-resource languages

TL;DR

This paper demonstrates that unsupervised sequence segmentation can be effectively transferred to extremely low-resource languages through multilingual pre-training of a Masked Segmental Language Model, especially benefiting small datasets and zero-shot scenarios.

Contribution

It introduces a method for transferring unsupervised segmentation capabilities to low-resource languages via multilingual pre-training on typologically similar languages.

Findings

Multilingual pre-training improves segmentation over monolingual models in most settings.

The approach achieves a zero-shot F1 score of 20.6.

Performance remains robust across various small dataset sizes.

Abstract

We show that unsupervised sequence-segmentation performance can be transferred to extremely low-resource languages by pre-training a Masked Segmental Language Model (Downey et al., 2021) multilingually. Further, we show that this transfer can be achieved by training over a collection of low-resource languages that are typologically similar (but phylogenetically unrelated) to the target language. In our experiments, we transfer from a collection of 10 Indigenous American languages (AmericasNLP, Mager et al., 2021) to K'iche', a Mayan language. We compare our multilingual model to a monolingual (from-scratch) baseline, as well as a model pre-trained on Quechua only. We show that the multilingual pre-trained approach yields consistent segmentation quality across target dataset sizes, exceeding the monolingual baseline in 6/10 experimental settings. Our model yields especially strong results at small target sizes, including a zero-shot performance of 20.6 F1. These results have promising implications for low-resource NLP pipelines involving human-like linguistic units, such as the sparse transcription framework proposed by Bird (2020).