Denoising Word Embeddings by Averaging in a Shared Space

TL;DR

This paper presents a novel method for enhancing word embeddings by averaging multiple models in a shared space, leading to more stable and improved representations, especially for rare words.

Contribution

The authors propose a shared space averaging technique using Generalized Procrustes Analysis to fuse independently trained embeddings, improving their quality and stability.

Findings

Consistent improvements over raw models and simple averaging.

Enhanced performance on rare word evaluations.

More stable and reliable word representations.

Abstract

We introduce a new approach for smoothing and improving the quality of word embeddings. We consider a method of fusing word embeddings that were trained on the same corpus but with different initializations. We project all the models to a shared vector space using an efficient implementation of the Generalized Procrustes Analysis (GPA) procedure, previously used in multilingual word translation. Our word representation demonstrates consistent improvements over the raw models as well as their simplistic average, on a range of tasks. As the new representations are more stable and reliable, there is a noticeable improvement in rare word evaluations.