Polyglot Neural Language Models: A Case Study in Cross-Lingual Phonetic Representation Learning

TL;DR

This paper presents polyglot neural language models trained across multiple languages to improve phonetic sequence modeling, demonstrating better generalization and higher quality representations than monolingual models.

Contribution

The introduction of polyglot neural language models that leverage shared representations and typological information for cross-lingual phonetic modeling is novel.

Findings

Polyglot models outperform monolingual models on held-out perplexity.

Polyglot phonetic features are of higher quality than monolingual ones.

Models generalize better to unseen data.

Abstract

We introduce polyglot language models, recurrent neural network models trained to predict symbol sequences in many different languages using shared representations of symbols and conditioning on typological information about the language to be predicted. We apply these to the problem of modeling phone sequences---a domain in which universal symbol inventories and cross-linguistically shared feature representations are a natural fit. Intrinsic evaluation on held-out perplexity, qualitative analysis of the learned representations, and extrinsic evaluation in two downstream applications that make use of phonetic features show (i) that polyglot models better generalize to held-out data than comparable monolingual models and (ii) that polyglot phonetic feature representations are of higher quality than those learned monolingually.