RuSentEval: Linguistic Source, Encoder Force!

TL;DR

This paper introduces RuSentEval, a comprehensive set of probing tasks for Russian, to analyze how multilingual transformers learn linguistic properties across Russian and English, revealing surprising insights about language representations.

Contribution

It presents RuSentEval with new probing tasks for Russian and compares linguistic property learning in multilingual transformers across Russian and English.

Findings

Some linguistic properties are learned similarly across languages.

Results challenge common assumptions about language-specific representations.

The study uncovers unexpected patterns in how models encode linguistic features.

Abstract

The success of pre-trained transformer language models has brought a great deal of interest on how these models work, and what they learn about language. However, prior research in the field is mainly devoted to English, and little is known regarding other languages. To this end, we introduce RuSentEval, an enhanced set of 14 probing tasks for Russian, including ones that have not been explored yet. We apply a combination of complementary probing methods to explore the distribution of various linguistic properties in five multilingual transformers for two typologically contrasting languages -- Russian and English. Our results provide intriguing findings that contradict the common understanding of how linguistic knowledge is represented, and demonstrate that some properties are learned in a similar manner despite the language differences.