Same Neurons, Different Languages: Probing Morphosyntax in Multilingual Pre-trained Models

TL;DR

This paper investigates whether multilingual pre-trained models encode morphosyntactic information in the same neurons across different languages, revealing significant cross-lingual neuron overlap influenced by language similarity and data size.

Contribution

It provides the first large-scale empirical analysis of neuron-level encoding of morphosyntax across 43 languages, highlighting shared neural representations.

Findings

Significant cross-lingual neuron overlap exists.

Overlap varies with language proximity and data size.

Morphosyntactic categories differ in neuron sharing.

Abstract

The success of multilingual pre-trained models is underpinned by their ability to learn representations shared by multiple languages even in absence of any explicit supervision. However, it remains unclear how these models learn to generalise across languages. In this work, we conjecture that multilingual pre-trained models can derive language-universal abstractions about grammar. In particular, we investigate whether morphosyntactic information is encoded in the same subset of neurons in different languages. We conduct the first large-scale empirical study over 43 languages and 14 morphosyntactic categories with a state-of-the-art neuron-level probe. Our findings show that the cross-lingual overlap between neurons is significant, but its extent may vary across categories and depends on language proximity and pre-training data size.