Consistent Alignment of Word Embedding Models

TL;DR

This paper presents a method to align different word embedding models in a shared space using synthetic data, improving the recovery of local neighborhood embeddings and capturing linguistic relationships more effectively.

Contribution

It introduces a novel alignment technique leveraging synthetic data to enhance the consistency of word embedding models across different instances or types.

Findings

Significant improvement in local neighborhood embedding recovery

Effective alignment of different word embedding models

Enhanced capture of linguistic relationships

Abstract

Word embedding models offer continuous vector representations that can capture rich contextual semantics based on their word co-occurrence patterns. While these word vectors can provide very effective features used in many NLP tasks such as clustering similar words and inferring learning relationships, many challenges and open research questions remain. In this paper, we propose a solution that aligns variations of the same model (or different models) in a joint low-dimensional latent space leveraging carefully generated synthetic data points. This generative process is inspired by the observation that a variety of linguistic relationships is captured by simple linear operations in embedded space. We demonstrate that our approach can lead to substantial improvements in recovering embeddings of local neighborhoods.