A recurrent neural network without chaos

TL;DR

This paper presents a simple gated RNN that matches the performance of complex models like LSTMs on language modeling tasks, with predictable and non-chaotic dynamics, unlike traditional gated architectures.

Contribution

Introduces a straightforward gated RNN with comparable performance to LSTMs and GRUs, demonstrating non-chaotic, predictable dynamics.

Findings

Achieves similar performance to LSTMs and GRUs on language modeling.

Exhibits simple, predictable, and non-chaotic dynamics.

Contrasts with standard gated architectures that show chaotic behavior.

Abstract

We introduce an exceptionally simple gated recurrent neural network (RNN) that achieves performance comparable to well-known gated architectures, such as LSTMs and GRUs, on the word-level language modeling task. We prove that our model has simple, predicable and non-chaotic dynamics. This stands in stark contrast to more standard gated architectures, whose underlying dynamical systems exhibit chaotic behavior.