A multilabel approach to morphosyntactic probing

TL;DR

This paper presents a multilabel probing task to evaluate how well multilingual BERT captures morphosyntactic features across diverse languages, revealing shared linguistic properties and transfer capabilities.

Contribution

It introduces a novel multilabel probing method for assessing morphosyntactic representations in multilingual models, demonstrating its effectiveness across multiple languages and in zero-shot transfer scenarios.

Findings

Multilingual BERT captures many morphosyntactic features effectively.

Probes perform well on recognizing nouns across languages.

Shared linguistic properties are revealed through probing.

Abstract

We introduce a multilabel probing task to assess the morphosyntactic representations of word embeddings from multilingual language models. We demonstrate this task with multilingual BERT (Devlin et al., 2018), training probes for seven typologically diverse languages of varying morphological complexity: Afrikaans, Croatian, Finnish, Hebrew, Korean, Spanish, and Turkish. Through this simple but robust paradigm, we show that multilingual BERT renders many morphosyntactic features easily and simultaneously extractable (e.g., gender, grammatical case, pronominal type). We further evaluate the probes on six "held-out" languages in a zero-shot transfer setting: Arabic, Chinese, Marathi, Slovenian, Tagalog, and Yoruba. This style of probing has the added benefit of revealing the linguistic properties that language models recognize as being shared across languages. For instance, the probes performed well on recognizing nouns in the held-out languages, suggesting that multilingual BERT has a conception of noun-hood that transcends individual languages; yet, the same was not true of adjectives.