Empirical Analysis of the Inductive Bias of Recurrent Neural Networks by Discrete Fourier Transform of Output Sequences

TL;DR

This paper investigates the inherent generalization tendencies of RNNs by analyzing the frequency characteristics of their output sequences using Fourier transform, revealing different biases among LSTM, GRU, and Elman RNNs.

Contribution

It introduces a novel frequency domain analysis method to directly measure RNNs' inductive bias regarding output sequence frequency patterns.

Findings

LSTM and GRU favor low-frequency output patterns

Elman RNN tends to learn high-frequency output changes

Inductive bias varies with network depth and size

Abstract

A unique feature of Recurrent Neural Networks (RNNs) is that it incrementally processes input sequences. In this research, we aim to uncover the inherent generalization properties, i.e., inductive bias, of RNNs with respect to how frequently RNNs switch the outputs through time steps in the sequence classification task, which we call output sequence frequency. Previous work analyzed inductive bias by training models with a few synthetic data and comparing the model's generalization with candidate generalization patterns. However, when examining the output sequence frequency, previous methods cannot be directly applied since enumerating candidate patterns is computationally difficult for longer sequences. To this end, we propose to directly calculate the output sequence frequency for each model by regarding the outputs of the model as discrete-time signals and applying frequency domain analysis. Experimental results showed that Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU) have an inductive bias towards lower-frequency patterns, while Elman RNN tends to learn patterns in which the output changes at high frequencies. We also found that the inductive bias of LSTM and GRU varies with the number of layers and the size of hidden layers.