A Comprehensive Survey on Word Representation Models: From Classical to State-Of-The-Art Word Representation Language Models

TL;DR

This survey comprehensively reviews the evolution of word representation models in NLP, from classical methods to modern state-of-the-art language models, highlighting their design, capabilities, and applications.

Contribution

It provides a detailed comparison of various text representation techniques and discusses the integration of these models with machine learning algorithms for NLP tasks.

Findings

Modern language models effectively capture semantic information.

Word embeddings enhance NLP task performance.

Survey covers classical to state-of-the-art models.

Abstract

Word representation has always been an important research area in the history of natural language processing (NLP). Understanding such complex text data is imperative, given that it is rich in information and can be used widely across various applications. In this survey, we explore different word representation models and its power of expression, from the classical to modern-day state-of-the-art word representation language models (LMS). We describe a variety of text representation methods, and model designs have blossomed in the context of NLP, including SOTA LMs. These models can transform large volumes of text into effective vector representations capturing the same semantic information. Further, such representations can be utilized by various machine learning (ML) algorithms for a variety of NLP related tasks. In the end, this survey briefly discusses the commonly used ML and DL based classifiers, evaluation metrics and the applications of these word embeddings in different NLP tasks.