Unsupervised Alignment of Embeddings with Wasserstein Procrustes

TL;DR

This paper introduces a novel unsupervised method for aligning high-dimensional embeddings using Wasserstein Procrustes, outperforming adversarial approaches in unsupervised word translation tasks.

Contribution

It proposes a joint estimation approach with convex relaxation initialization for embedding alignment, offering a scalable and more efficient alternative to adversarial training.

Findings

Achieves state-of-the-art results in unsupervised word translation

Requires less computational resources than existing methods

Effective in high-dimensional embedding alignment

Abstract

We consider the task of aligning two sets of points in high dimension, which has many applications in natural language processing and computer vision. As an example, it was recently shown that it is possible to infer a bilingual lexicon, without supervised data, by aligning word embeddings trained on monolingual data. These recent advances are based on adversarial training to learn the mapping between the two embeddings. In this paper, we propose to use an alternative formulation, based on the joint estimation of an orthogonal matrix and a permutation matrix. While this problem is not convex, we propose to initialize our optimization algorithm by using a convex relaxation, traditionally considered for the graph isomorphism problem. We propose a stochastic algorithm to minimize our cost function on large scale problems. Finally, we evaluate our method on the problem of unsupervised word translation, by aligning word embeddings trained on monolingual data. On this task, our method obtains state of the art results, while requiring less computational resources than competing approaches.