An Isotropy Analysis in the Multilingual BERT Embedding Space

TL;DR

This paper investigates the anisotropic properties of multilingual BERT embeddings, revealing that it lacks outlier dimensions but is highly anisotropic, and that increasing isotropy enhances its performance.

Contribution

It provides the first detailed analysis of anisotropy in multilingual BERT, showing how isotropy improvements can boost multilingual representation quality.

Findings

Multilingual BERT has no outlier dimensions but is highly anisotropic.

Increasing isotropy improves multilingual BERT's performance.

Embedding spaces across languages are structurally similar despite anisotropy.

Abstract

Several studies have explored various advantages of multilingual pre-trained models (such as multilingual BERT) in capturing shared linguistic knowledge. However, less attention has been paid to their limitations. In this paper, we investigate the multilingual BERT for two known issues of the monolingual models: anisotropic embedding space and outlier dimensions. We show that, unlike its monolingual counterpart, the multilingual BERT model exhibits no outlier dimension in its representations while it has a highly anisotropic space. There are a few dimensions in the monolingual BERT with high contributions to the anisotropic distribution. However, we observe no such dimensions in the multilingual BERT. Furthermore, our experimental results demonstrate that increasing the isotropy of multilingual space can significantly improve its representation power and performance, similarly to what had been observed for monolingual CWRs on semantic similarity tasks. Our analysis indicates that, despite having different degenerated directions, the embedding spaces in various languages tend to be partially similar with respect to their structures.