Learnability Window in Gated Recurrent Neural Networks

TL;DR

This paper presents a statistical theory for understanding the limits of temporal learning in gated RNNs, linking gating mechanisms and optimizer effects to the maximum recoverable temporal horizon.

Contribution

It introduces the effective learning rate envelope and derives explicit scaling laws for the learnability window under different noise conditions.

Findings

Slower envelope decay increases the learnability window.

Heavy-tailed noise reduces the temporal horizon due to weaker concentration.

Experiments confirm the theoretical predictions across architectures and optimizers.

Abstract

We develop a statistical theory of temporal learnability in recurrent neural networks, quantifying the maximal temporal horizon $\mathcal{H}_N$ over which gradient-based learning can recover lag-dependent structure at finite sample size $N$. The theory is built on the effective learning rate envelope $f(\ell)$, a functional that captures how gating mechanisms and adaptive optimizers jointly shape the coupling between state-space transport and parameter updates during Backpropagation Through Time. Under heavy-tailed ($\alpha$-stable) fluctuations, where empirical averages concentrate at rate $N^{-1/\kappa_\alpha}$ with $\kappa_\alpha = \alpha/(\alpha-1)$, the interplay between envelope decay and statistical concentration yields explicit scaling laws for the growth of $\mathcal{H}_N$: logarithmic, polynomial, and exponential temporal learning regimes emerge according to the decay law of $f(\ell)$. These results identify the envelope decay as the key determinant of temporal learnability: slower attenuation of $f(\ell)$ enlarges the learnability window $\mathcal{H}_N$, while heavy-tailed noise compresses temporal horizons by weakening statistical concentration. Experiments across multiple gated architectures and optimizers corroborate these structural predictions.