Language Models with Pre-Trained (GloVe) Word Embeddings

TL;DR

This paper explores training a language model using GloVe pre-trained word embeddings combined with a GRU-based RNN architecture, offering an alternative to LSTM-based models.

Contribution

It introduces a language model implementation utilizing GloVe embeddings and GRU units, differing from typical LSTM-based approaches.

Findings

Demonstrates feasibility of GloVe embeddings with GRU for language modeling

Provides implementation details for training such models

Suggests potential advantages over LSTM-based models

Abstract

In this work we implement a training of a Language Model (LM), using Recurrent Neural Network (RNN) and GloVe word embeddings, introduced by Pennigton et al. in [1]. The implementation is following the general idea of training RNNs for LM tasks presented in [2], but is rather using Gated Recurrent Unit (GRU) [3] for a memory cell, and not the more commonly used LSTM [4].