# Sequential Recurrent Neural Networks for Language Modeling

## TL;DR

This paper introduces a novel neural network architecture that combines the explicit sequential history modeling of feedforward networks with the dynamic context capturing of recurrent networks, significantly improving language modeling performance.

## Contribution

It proposes a new architecture that integrates explicit word history with recurrent context, bridging FNN and RNN approaches for better language modeling.

## Key findings

- Significant perplexity reduction on PTB and LTCB datasets.
- Outperforms state-of-the-art FNN and RNN language models.
- Demonstrates the effectiveness of combined sequential and recurrent context.

## Abstract

Feedforward Neural Network (FNN)-based language models estimate the probability of the next word based on the history of the last N words, whereas Recurrent Neural Networks (RNN) perform the same task based only on the last word and some context information that cycles in the network. This paper presents a novel approach, which bridges the gap between these two categories of networks. In particular, we propose an architecture which takes advantage of the explicit, sequential enumeration of the word history in FNN structure while enhancing each word representation at the projection layer through recurrent context information that evolves in the network. The context integration is performed using an additional word-dependent weight matrix that is also learned during the training. Extensive experiments conducted on the Penn Treebank (PTB) and the Large Text Compression Benchmark (LTCB) corpus showed a significant reduction of the perplexity when compared to state-of-the-art feedforward as well as recurrent neural network architectures.