Syllable-level Neural Language Model for Agglutinative Language

TL;DR

This paper introduces a syllable- and morpheme-based embedding method for agglutinative language models, significantly reducing out-of-vocabulary issues and outperforming character-level models in perplexity and input prediction, with commercial success.

Contribution

It presents a novel embedding approach leveraging syllables and morphemes to improve language modeling for agglutinative languages, outperforming existing methods.

Findings

16.87 perplexity reduction over character-level models

Achieved state-of-the-art in Key Stroke Saving

Model has been commercialized

Abstract

Language models for agglutinative languages have always been hindered in past due to myriad of agglutinations possible to any given word through various affixes. We propose a method to diminish the problem of out-of-vocabulary words by introducing an embedding derived from syllables and morphemes which leverages the agglutinative property. Our model outperforms character-level embedding in perplexity by 16.87 with 9.50M parameters. Proposed method achieves state of the art performance over existing input prediction methods in terms of Key Stroke Saving and has been commercialized.