BERT Cannot Align Characters

TL;DR

This paper investigates BERT's ability to perform character-level alignment across languages, revealing that its effectiveness depends on language similarity and proximity, with better results for closely related languages.

Contribution

The study demonstrates that BERT's character-level alignment ability is limited and varies with language similarity, highlighting the importance of language proximity in cross-lingual tasks.

Findings

BERT aligns closely related languages better at the character level.

Alignment performance decreases as linguistic distance increases.

Language similarity influences BERT's cross-lingual alignment effectiveness.

Abstract

In previous work, it has been shown that BERT can adequately align cross-lingual sentences on the word level. Here we investigate whether BERT can also operate as a char-level aligner. The languages examined are English, Fake-English, German and Greek. We show that the closer two languages are, the better BERT can align them on the character level. BERT indeed works well in English to Fake-English alignment, but this does not generalize to natural languages to the same extent. Nevertheless, the proximity of two languages does seem to be a factor. English is more related to German than to Greek and this is reflected in how well BERT aligns them; English to German is better than English to Greek. We examine multiple setups and show that the similarity matrices for natural languages show weaker relations the further apart two languages are.