A Quantifier-Reversal Approximation Paradigm for Recurrent Neural Networks

TL;DR

This paper proposes a new approximation paradigm for RNNs that achieves arbitrary accuracy by increasing runtime rather than network size, using fixed weights and topology, suitable for memory-constrained hardware.

Contribution

It introduces a quantifier-reversal approximation method for RNNs, enabling fixed-weight networks to approximate functions to arbitrary precision through extended runtime.

Findings

RNNs can approximate univariate polynomials with fixed weights.

Approximation error decays exponentially with runtime.

Network size depends on polynomial degree, not error tolerance.

Abstract

Classical neural network approximation results take the form: for every function $f$ and every error tolerance $\epsilon > 0$, one constructs a neural network whose architecture and weights depend on $\epsilon$. This paper introduces a fundamentally different approximation paradigm that reverses this quantifier order. For each target function $f$, we construct a single recurrent neural network (RNN) with fixed topology and fixed weights that approximates $f$ to within any prescribed tolerance $\epsilon > 0$ when run for sufficiently many time steps. The key mechanism enabling this quantifier reversal is temporal computation combined with weight sharing: rather than increasing network depth, the approximation error is reduced solely by running the RNN longer. This yields exponentially decaying approximation error as a function of runtime while requiring storage of only a small, fixed set of weights. Such architectures are appealing for hardware implementations where memory is scarce and runtime is comparatively inexpensive. To initiate the systematic development of this novel approximation paradigm, we focus on univariate polynomials. Our RNN constructions emulate the structural calculus underlying deep feed-forward ReLU network approximation theory -- parallelization, linear combinations, affine transformations, and, most importantly, a clocked mechanism that realizes function composition within a single recurrent architecture. The resulting RNNs have size independent of the error tolerance $\epsilon$ and hidden-state dimension linear in the degree of the polynomial.