Correspondence Analysis and PMI-Based Word Embeddings: A Comparative Study

TL;DR

This paper explores the mathematical connection between correspondence analysis and PMI-based word embeddings, proposing variants that improve performance and comparing them with BERT across multiple benchmarks.

Contribution

It establishes a formal link between CA and PMI-based embeddings and introduces ROOT-CA and ROOTROOT-CA variants that enhance embedding quality.

Findings

ROOT-CA and ROOTROOT-CA outperform standard PMI methods

Variants achieve results competitive with BERT

Performance depends on handling extreme matrix values

Abstract

Popular word embedding methods such as GloVe and Word2Vec are related to the factorization of the pointwise mutual information (PMI) matrix. In this paper, we establish a formal connection between correspondence analysis (CA) and PMI-based word embedding methods. CA is a dimensionality reduction method that uses singular value decomposition (SVD), and we show that CA is mathematically close to the weighted factorization of the PMI matrix. We further introduce variants of CA for word-context matrices, namely CA applied after a square-root transformation (ROOT-CA) and after a fourth-root transformation (ROOTROOT-CA). We analyze the performance of these methods and examine how their success or failure is influenced by extreme values in the decomposed matrix. Although our primary focus is on traditionalstatic word embedding methods, we also include a comparison with a transformer-based encoder (BERT) to situate the results relative to contextual embeddings. Empirical evaluations across multiple corpora and word-similarity benchmarks show that ROOT-CA and ROOTROOT-CA perform slightly better overall than standard PMI-based methods and achieve results competitive with BERT.