A Masked Segmental Language Model for Unsupervised Natural Language Segmentation

TL;DR

This paper introduces a Masked Segmental Language Model (MSLM) using span-masking transformers for unsupervised language segmentation, outperforming recurrent models on Chinese and matching them on English, addressing challenges in morphologically complex and low-resource languages.

Contribution

The paper presents a novel span-masking transformer-based segmentation model that improves unsupervised language segmentation, especially for morphologically complex and resource-scarce languages.

Findings

Outperforms Recurrent SLMs on Chinese segmentation quality.

Performs comparably to Recurrent models on English.

Effectively handles phonemic writing systems.

Abstract

Segmentation remains an important preprocessing step both in languages where "words" or other important syntactic/semantic units (like morphemes) are not clearly delineated by white space, as well as when dealing with continuous speech data, where there is often no meaningful pause between words. Near-perfect supervised methods have been developed for use in resource-rich languages such as Chinese, but many of the world's languages are both morphologically complex, and have no large dataset of "gold" segmentations into meaningful units. To solve this problem, we propose a new type of Segmental Language Model (Sun and Deng, 2018; Kawakami et al., 2019; Wang et al., 2021) for use in both unsupervised and lightly supervised segmentation tasks. We introduce a Masked Segmental Language Model (MSLM) built on a span-masking transformer architecture, harnessing the power of a bi-directional masked modeling context and attention. In a series of experiments, our model consistently outperforms Recurrent SLMs on Chinese (PKU Corpus) in segmentation quality, and performs similarly to the Recurrent model on English (PTB). We conclude by discussing the different challenges posed in segmenting phonemic-type writing systems.