Blind signal decomposition of various word embeddings based on join and individual variance explained

TL;DR

This paper introduces a joint signal decomposition method, JIVE, to analyze and fuse different trained word embeddings, revealing their shared and unique features and improving sentiment analysis performance.

Contribution

The paper applies JIVE to decompose word embeddings into joint and individual components, offering new insights and methods for enhancing NLP tasks.

Findings

Decomposition improves sentiment analysis accuracy.

Mapping to joint components enhances lower-performing embeddings.

Concatenating components yields better model performance.

Abstract

In recent years, natural language processing (NLP) has become one of the most important areas with various applications in human's life. As the most fundamental task, the field of word embedding still requires more attention and research. Currently, existing works about word embedding are focusing on proposing novel embedding algorithms and dimension reduction techniques on well-trained word embeddings. In this paper, we propose to use a novel joint signal separation method - JIVE to jointly decompose various trained word embeddings into joint and individual components. Through this decomposition framework, we can easily investigate the similarity and difference among different word embeddings. We conducted extensive empirical study on word2vec, FastText and GLoVE trained on different corpus and with different dimensions. We compared the performance of different decomposed components based on sentiment analysis on Twitter and Stanford sentiment treebank. We found that by mapping different word embeddings into the joint component, sentiment performance can be greatly improved for the original word embeddings with lower performance. Moreover, we found that by concatenating different components together, the same model can achieve better performance. These findings provide great insights into the word embeddings and our work offer a new of generating word embeddings by fusing.