A Survey On Neural Word Embeddings

TL;DR

This survey comprehensively reviews neural word embeddings, covering their theoretical foundations, various types, and evaluation benchmarks, highlighting their transformative impact on NLP tasks.

Contribution

It provides an extensive overview of neural word embeddings, including recent developments like contextual representations, and discusses evaluation methods and performance results.

Findings

Neural embeddings significantly improve NLP task performance.

Various types of embeddings capture different semantic aspects.

Benchmark datasets are essential for evaluating embedding quality.

Abstract

Understanding human language has been a sub-challenge on the way of intelligent machines. The study of meaning in natural language processing (NLP) relies on the distributional hypothesis where language elements get meaning from the words that co-occur within contexts. The revolutionary idea of distributed representation for a concept is close to the working of a human mind in that the meaning of a word is spread across several neurons, and a loss of activation will only slightly affect the memory retrieval process. Neural word embeddings transformed the whole field of NLP by introducing substantial improvements in all NLP tasks. In this survey, we provide a comprehensive literature review on neural word embeddings. We give theoretical foundations and describe existing work by an interplay between word embeddings and language modelling. We provide broad coverage on neural word embeddings, including early word embeddings, embeddings targeting specific semantic relations, sense embeddings, morpheme embeddings, and finally, contextual representations. Finally, we describe benchmark datasets in word embeddings' performance evaluation and downstream tasks along with the performance results of/due to word embeddings.