Morph Call: Probing Morphosyntactic Content of Multilingual Transformers

TL;DR

This paper introduces Morph Call, a comprehensive suite of probing tasks for analyzing the morphosyntactic knowledge of multilingual transformer models across four Indo-European languages, including effects of fine-tuning.

Contribution

It presents a novel set of 46 probing tasks, including sentence perturbation detection, and applies multiple introspection techniques to analyze multilingual transformers and their distilled versions.

Findings

Fine-tuning can both improve and reduce morphosyntactic knowledge.

Fine-tuning alters the distribution of morphosyntactic information within models.

Distilled models retain significant morphosyntactic content.

Abstract

The outstanding performance of transformer-based language models on a great variety of NLP and NLU tasks has stimulated interest in exploring their inner workings. Recent research has focused primarily on higher-level and complex linguistic phenomena such as syntax, semantics, world knowledge, and common sense. The majority of the studies are anglocentric, and little remains known regarding other languages, precisely their morphosyntactic properties. To this end, our work presents Morph Call, a suite of 46 probing tasks for four Indo-European languages of different morphology: English, French, German and Russian. We propose a new type of probing task based on the detection of guided sentence perturbations. We use a combination of neuron-, layer- and representation-level introspection techniques to analyze the morphosyntactic content of four multilingual transformers, including their less explored distilled versions. Besides, we examine how fine-tuning for POS-tagging affects the model knowledge. The results show that fine-tuning can improve and decrease the probing performance and change how morphosyntactic knowledge is distributed across the model. The code and data are publicly available, and we hope to fill the gaps in the less studied aspect of transformers.