A Technical Note on the Architectural Effects on Maximum Dependency Lengths of Recurrent Neural Networks

TL;DR

This paper introduces a methodology to measure the maximum dependency length in RNNs and examines how architectural modifications influence this length across different RNN variants.

Contribution

It provides a systematic approach to quantify dependency lengths and analyzes the impact of architectural choices on RNN memory capabilities.

Findings

Architectural changes significantly affect maximum dependency lengths.

Gated units like GRU and LSTM show different dependency behaviors than traditional RNNs.

Increasing layers and neurons can extend the dependency length capacity.

Abstract

This work proposes a methodology for determining the maximum dependency length of a recurrent neural network (RNN), and then studies the effects of architectural changes, including the number and neuron count of layers, on the maximum dependency lengths of traditional RNN, gated recurrent unit (GRU), and long-short term memory (LSTM) models.