Word Emdeddings through Hellinger PCA

TL;DR

This paper introduces a simplified method for generating word embeddings using Hellinger PCA on co-occurrence matrices, achieving comparable or better results than neural models on NLP tasks.

Contribution

It proposes a novel, less complex approach to word embedding creation via Hellinger PCA, bypassing neural network training.

Findings

Hellinger PCA embeddings perform well on NER and sentiment analysis.

The method is faster and easier to implement than neural models.

Task-specific adaptation of embeddings is feasible with this approach.

Abstract

Word embeddings resulting from neural language models have been shown to be successful for a large variety of NLP tasks. However, such architecture might be difficult to train and time-consuming. Instead, we propose to drastically simplify the word embeddings computation through a Hellinger PCA of the word co-occurence matrix. We compare those new word embeddings with some well-known embeddings on NER and movie review tasks and show that we can reach similar or even better performance. Although deep learning is not really necessary for generating good word embeddings, we show that it can provide an easy way to adapt embeddings to specific tasks.