Canonical and Surface Morphological Segmentation for Nguni Languages

TL;DR

This paper compares supervised and unsupervised models for morphological segmentation of Nguni languages, showing that transformers excel in canonical segmentation and CRFs in surface segmentation, aiding NLP development for these languages.

Contribution

It introduces and evaluates models for canonical and surface segmentation in Nguni languages, highlighting the effectiveness of transformers and CRFs in these tasks.

Findings

Transformers outperform LSTMs in canonical segmentation with 72.5% F1.

CRFs outperform LSTM-CRFs in surface segmentation with 97.1% F1.

Unsupervised methods like Morfessor and language models perform poorly on these languages.

Abstract

Morphological Segmentation involves decomposing words into morphemes, the smallest meaning-bearing units of language. This is an important NLP task for morphologically-rich agglutinative languages such as the Southern African Nguni language group. In this paper, we investigate supervised and unsupervised models for two variants of morphological segmentation: canonical and surface segmentation. We train sequence-to-sequence models for canonical segmentation, where the underlying morphemes may not be equal to the surface form of the word, and Conditional Random Fields (CRF) for surface segmentation. Transformers outperform LSTMs with attention on canonical segmentation, obtaining an average F1 score of 72.5% across 4 languages. Feature-based CRFs outperform bidirectional LSTM-CRFs to obtain an average of 97.1% F1 on surface segmentation. In the unsupervised setting, an entropy-based approach using a character-level LSTM language model fails to outperforms a Morfessor baseline, while on some of the languages neither approach performs much better than a random baseline. We hope that the high performance of the supervised segmentation models will help to facilitate the development of better NLP tools for Nguni languages.