Are Word Embedding Methods Stable and Should We Care About It?

TL;DR

This paper investigates the stability of popular word embedding methods across different datasets and parameters, revealing that fastText is the most stable among Word2Vec, GloVe, and fastText, and examines implications for downstream tasks.

Contribution

It introduces a systematic stability measurement for WEMs using intrinsic evaluation and analyzes how stability varies with parameters and datasets.

Findings

fastText is the most stable WEM among the three studied

Stability varies significantly with training parameters and datasets

Stability impacts downstream tasks like clustering and POS tagging

Abstract

A representation learning method is considered stable if it consistently generates similar representation of the given data across multiple runs. Word Embedding Methods (WEMs) are a class of representation learning methods that generate dense vector representation for each word in the given text data. The central idea of this paper is to explore the stability measurement of WEMs using intrinsic evaluation based on word similarity. We experiment with three popular WEMs: Word2Vec, GloVe, and fastText. For stability measurement, we investigate the effect of five parameters involved in training these models. We perform experiments using four real-world datasets from different domains: Wikipedia, News, Song lyrics, and European parliament proceedings. We also observe the effect of WEM stability on three downstream tasks: Clustering, POS tagging, and Fairness evaluation. Our experiments indicate that amongst the three WEMs, fastText is the most stable, followed by GloVe and Word2Vec.