Low-Rank Approximation of Matrices for PMI-based Word Embeddings

TL;DR

This paper empirically compares low-rank approximation methods for PMI-based word embeddings, finding that truncated SVD yields the best performance on similarity and analogy tasks.

Contribution

It provides an empirical evaluation of SVD, NMF, and QR for low-rank approximation in PMI-based embeddings, highlighting SVD's superior effectiveness.

Findings

Truncated SVD outperforms NMF and QR in downstream tasks.

Word vectors from SVD achieve higher similarity and analogy scores.

Empirical evidence supports SVD as the preferred method for PMI matrix approximation.

Abstract

We perform an empirical evaluation of several methods of low-rank approximation in the problem of obtaining PMI-based word embeddings. All word vectors were trained on parts of a large corpus extracted from English Wikipedia (enwik9) which was divided into two equal-sized datasets, from which PMI matrices were obtained. A repeated measures design was used in assigning a method of low-rank approximation (SVD, NMF, QR) and dimensionality of the vectors (250, 500) to each of the PMI matrix replicates. Our experiments show that word vectors obtained from the truncated SVD achieve the best performance on two downstream tasks, similarity and analogy, compare to the other two low-rank approximation methods.