Are Girls Neko or Sh\=ojo? Cross-Lingual Alignment of Non-Isomorphic Embeddings with Iterative Normalization

TL;DR

This paper introduces Iterative Normalization, a method to improve cross-lingual word embeddings for non-isomorphic language pairs by normalizing embeddings, significantly enhancing translation accuracy especially for challenging pairs like English-Japanese.

Contribution

The paper proposes a novel normalization technique that facilitates better orthogonal alignment of non-isomorphic embeddings, improving multilingual NLP performance.

Findings

Significant accuracy improvements on English-Japanese translation

Consistent enhancement across three CLWE methods

Largest gain from 2% to 44% accuracy on Japanese-English pair

Abstract

Cross-lingual word embeddings (CLWE) underlie many multilingual natural language processing systems, often through orthogonal transformations of pre-trained monolingual embeddings. However, orthogonal mapping only works on language pairs whose embeddings are naturally isomorphic. For non-isomorphic pairs, our method (Iterative Normalization) transforms monolingual embeddings to make orthogonal alignment easier by simultaneously enforcing that (1) individual word vectors are unit length, and (2) each language's average vector is zero. Iterative Normalization consistently improves word translation accuracy of three CLWE methods, with the largest improvement observed on English-Japanese (from 2% to 44% test accuracy).