Evaluating Word Embeddings with Categorical Modularity

TL;DR

This paper introduces categorical modularity, a new low-resource intrinsic metric based on graph modularity, to evaluate word embedding quality across languages and models, showing correlations with downstream tasks.

Contribution

The paper proposes categorical modularity as a novel intrinsic evaluation metric for word embeddings, applicable across multiple languages and models, with demonstrated predictive power for downstream tasks.

Findings

Moderate to strong correlation with sentiment analysis and word similarity tasks.

Predictive of cross-lingual bilingual lexicon induction performance.

Provides insights into semantic information loss in embeddings.

Abstract

We introduce categorical modularity, a novel low-resource intrinsic metric to evaluate word embedding quality. Categorical modularity is a graph modularity metric based on the $k$-nearest neighbor graph constructed with embedding vectors of words from a fixed set of semantic categories, in which the goal is to measure the proportion of words that have nearest neighbors within the same categories. We use a core set of 500 words belonging to 59 neurobiologically motivated semantic categories in 29 languages and analyze three word embedding models per language (FastText, MUSE, and subs2vec). We find moderate to strong positive correlations between categorical modularity and performance on the monolingual tasks of sentiment analysis and word similarity calculation and on the cross-lingual task of bilingual lexicon induction both to and from English. Overall, we suggest that categorical modularity provides non-trivial predictive information about downstream task performance, with breakdowns of correlations by model suggesting some meta-predictive properties about semantic information loss as well.