# Multiplicative Models for Recurrent Language Modeling

## TL;DR

This paper investigates multiplicative recurrent neural networks, especially mLSTM, for language modeling, demonstrating that shared second-order terms improve sequence generation by mitigating error accumulation.

## Contribution

The paper introduces new multiplicative RNN models with shared second-order terms and evaluates their effectiveness on character-level language modeling tasks.

## Key findings

- Shared parametrization enhances language modeling performance.
- Multiplicative models outperform traditional RNNs in sequence generation.
- Architectural improvements reduce error propagation in recurrent networks.

## Abstract

Recently, there has been interest in multiplicative recurrent neural networks for language modeling. Indeed, simple Recurrent Neural Networks (RNNs) encounter difficulties recovering from past mistakes when generating sequences due to high correlation between hidden states. These challenges can be mitigated by integrating second-order terms in the hidden-state update. One such model, multiplicative Long Short-Term Memory (mLSTM) is particularly interesting in its original formulation because of the sharing of its second-order term, referred to as the intermediate state. We explore these architectural improvements by introducing new models and testing them on character-level language modeling tasks. This allows us to establish the relevance of shared parametrization in recurrent language modeling.