Robust Unsupervised Cross-Lingual Word Embedding using Domain Flow Interpolation

TL;DR

This paper introduces a novel method for creating robust cross-lingual word embeddings by interpolating through intermediate pseudo-language spaces, improving stability and performance especially for distant language pairs.

Contribution

It proposes a domain flow interpolation technique that enhances the robustness of unsupervised cross-lingual embeddings over traditional adversarial methods.

Findings

Improved robustness in cross-lingual embedding training.

Better performance on bilingual dictionary induction.

Significant gains in cross-lingual natural language inference for distant languages.

Abstract

This paper investigates an unsupervised approach towards deriving a universal, cross-lingual word embedding space, where words with similar semantics from different languages are close to one another. Previous adversarial approaches have shown promising results in inducing cross-lingual word embedding without parallel data. However, the training stage shows instability for distant language pairs. Instead of mapping the source language space directly to the target language space, we propose to make use of a sequence of intermediate spaces for smooth bridging. Each intermediate space may be conceived as a pseudo-language space and is introduced via simple linear interpolation. This approach is modeled after domain flow in computer vision, but with a modified objective function. Experiments on intrinsic Bilingual Dictionary Induction tasks show that the proposed approach can improve the robustness of adversarial models with comparable and even better precision. Further experiments on the downstream task of Cross-Lingual Natural Language Inference show that the proposed model achieves significant performance improvement for distant language pairs in downstream tasks compared to state-of-the-art adversarial and non-adversarial models.