WOVe: Incorporating Word Order in GloVe Word Embeddings

TL;DR

This paper introduces WOVe, a method to incorporate word order into GloVe embeddings, significantly enhancing performance on analogy and similarity tasks.

Contribution

It proposes multiple approaches to embed word order into GloVe, improving its effectiveness in natural language understanding tasks.

Findings

WOVe outperforms GloVe on analogy completion by 36.34%

WOVe with concatenation increases word similarity accuracy by 2%

The methods significantly improve GloVe's performance on key NLP tasks

Abstract

Word vector representations open up new opportunities to extract useful information from unstructured text. Defining a word as a vector made it easy for the machine learning algorithms to understand a text and extract information from. Word vector representations have been used in many applications such word synonyms, word analogy, syntactic parsing, and many others. GloVe, based on word contexts and matrix vectorization, is an ef-fective vector-learning algorithm. It improves on previous vector-learning algorithms. However, the GloVe model fails to explicitly consider the order in which words appear within their contexts. In this paper, multiple methods of incorporating word order in GloVe word embeddings are proposed. Experimental results show that our Word Order Vector (WOVe) word embeddings approach outperforms unmodified GloVe on the natural lan-guage tasks of analogy completion and word similarity. WOVe with direct concatenation slightly outperformed GloVe on the word similarity task, increasing average rank by 2%. However, it greatly improved on the GloVe baseline on a word analogy task, achieving an average 36.34% improvement in accuracy.