Revisiting Neural Language Modelling with Syllables

TL;DR

This paper explores the use of syllables in neural language models across 20 languages, demonstrating that syllables outperform characters and other subword units in perplexity, with potential benefits for open-vocabulary generation.

Contribution

It introduces a rule-based and hyphenation tool approach for syllabification in multiple languages and compares syllables to other subword units, highlighting their advantages in language modelling.

Findings

Syllables outperform characters and morphemes in perplexity.

Syllabification methods are effective across diverse languages.

Syllables overlap with other subword units, revealing limitations and opportunities.

Abstract

Language modelling is regularly analysed at word, subword or character units, but syllables are seldom used. Syllables provide shorter sequences than characters, they can be extracted with rules, and their segmentation typically requires less specialised effort than identifying morphemes. We reconsider syllables for an open-vocabulary generation task in 20 languages. We use rule-based syllabification methods for five languages and address the rest with a hyphenation tool, which behaviour as syllable proxy is validated. With a comparable perplexity, we show that syllables outperform characters, annotated morphemes and unsupervised subwords. Finally, we also study the overlapping of syllables concerning other subword pieces and discuss some limitations and opportunities.