Probing Multilingual BERT for Genetic and Typological Signals

TL;DR

This paper investigates how multilingual BERT encodes genetic and geographic language signals, using it to infer language relationships, analyze factors influencing language distances, and measure diachronic meaning stability, advancing understanding of cross-lingual representations.

Contribution

It introduces methods to extract phylogenetic and typological signals from mBERT, evaluates language trees, and proposes a novel measure for diachronic meaning stability, enhancing interpretability of multilingual models.

Findings

Language distances from mBERT align closely with reference language trees.

Phylogenetic factors best explain language distance variations.

A new measure correlates with established linguistic stability rankings.

Abstract

We probe the layers in multilingual BERT (mBERT) for phylogenetic and geographic language signals across 100 languages and compute language distances based on the mBERT representations. We 1) employ the language distances to infer and evaluate language trees, finding that they are close to the reference family tree in terms of quartet tree distance, 2) perform distance matrix regression analysis, finding that the language distances can be best explained by phylogenetic and worst by structural factors and 3) present a novel measure for measuring diachronic meaning stability (based on cross-lingual representation variability) which correlates significantly with published ranked lists based on linguistic approaches. Our results contribute to the nascent field of typological interpretability of cross-lingual text representations.