From Phonology to Syntax: Unsupervised Linguistic Typology at Different Levels with Language Embeddings

TL;DR

This paper develops unsupervised language embeddings across phonology, morphology, and syntax for over 800 languages, enabling typological predictions and revealing how linguistic similarities are encoded in NLP models.

Contribution

It introduces a multilingual, unsupervised approach to learn language representations at multiple typological levels, facilitating typological classification and analysis.

Findings

Language embeddings encode typological similarities and differences.

High accuracy in predicting WALS features, even for unseen language families.

Distinct embeddings reflect phonological, morphological, and syntactic properties.

Abstract

A core part of linguistic typology is the classification of languages according to linguistic properties, such as those detailed in the World Atlas of Language Structure (WALS). Doing this manually is prohibitively time-consuming, which is in part evidenced by the fact that only 100 out of over 7,000 languages spoken in the world are fully covered in WALS. We learn distributed language representations, which can be used to predict typological properties on a massively multilingual scale. Additionally, quantitative and qualitative analyses of these language embeddings can tell us how language similarities are encoded in NLP models for tasks at different typological levels. The representations are learned in an unsupervised manner alongside tasks at three typological levels: phonology (grapheme-to-phoneme prediction, and phoneme reconstruction), morphology (morphological inflection), and syntax (part-of-speech tagging). We consider more than 800 languages and find significant differences in the language representations encoded, depending on the target task. For instance, although Norwegian Bokm{\aa}l and Danish are typologically close to one another, they are phonologically distant, which is reflected in their language embeddings growing relatively distant in a phonological task. We are also able to predict typological features in WALS with high accuracies, even for unseen language families.