A Survey on Neural Network Language Models

TL;DR

This survey reviews neural network language models in NLP, detailing their structures, improvements, datasets, tools, and future research directions, highlighting their advantages over traditional models.

Contribution

It provides a comprehensive overview of NNLMs, including their architectures, enhancements, datasets, and tools, and discusses future research avenues.

Findings

Summarizes major NNLM architectures and improvements.

Compares datasets and toolkits used in NNLM research.

Discusses future research directions in NNLMs.

Abstract

As the core component of Natural Language Processing (NLP) system, Language Model (LM) can provide word representation and probability indication of word sequences. Neural Network Language Models (NNLMs) overcome the curse of dimensionality and improve the performance of traditional LMs. A survey on NNLMs is performed in this paper. The structure of classic NNLMs is described firstly, and then some major improvements are introduced and analyzed. We summarize and compare corpora and toolkits of NNLMs. Further, some research directions of NNLMs are discussed.