Post-Processing of Word Representations via Variance Normalization and Dynamic Embedding

TL;DR

This paper introduces two novel post-processing techniques, PVN and PDE, that enhance word embeddings by normalizing variance and capturing sequence order, leading to improved NLP performance.

Contribution

The paper proposes two new post-processing methods, PVN and PDE, which improve word embeddings by normalizing variance and modeling sequence order, respectively.

Findings

PVN improves embedding quality by variance normalization.

PDE captures sequence order information in embeddings.

Combined PVN and PDE outperform baseline embeddings.

Abstract

Although embedded vector representations of words offer impressive performance on many natural language processing (NLP) applications, the information of ordered input sequences is lost to some extent if only context-based samples are used in the training. For further performance improvement, two new post-processing techniques, called post-processing via variance normalization (PVN) and post-processing via dynamic embedding (PDE), are proposed in this work. The PVN method normalizes the variance of principal components of word vectors while the PDE method learns orthogonal latent variables from ordered input sequences. The PVN and the PDE methods can be integrated to achieve better performance. We apply these post-processing techniques to two popular word embedding methods (i.e., word2vec and GloVe) to yield their post-processed representations. Extensive experiments are conducted to demonstrate the effectiveness of the proposed post-processing techniques.