Syllable-aware Neural Language Models: A Failure to Beat Character-aware Ones

TL;DR

This paper compares syllable-aware and character-aware neural language models, finding that syllable-aware models do not outperform character-aware ones in quality but are more parameter-efficient and faster to train.

Contribution

It demonstrates that syllable-aware models can match character-aware performance with fewer parameters and faster training, challenging assumptions about syllable segmentation benefits.

Findings

Syllable-aware models do not improve language modeling quality over character-aware models.

Syllable-aware models are 18%-33% smaller in parameters.

Syllable-aware models train 1.2-2.2 times faster.

Abstract

Syllabification does not seem to improve word-level RNN language modeling quality when compared to character-based segmentation. However, our best syllable-aware language model, achieving performance comparable to the competitive character-aware model, has 18%-33% fewer parameters and is trained 1.2-2.2 times faster.