Approximation Bounds for Recurrent Neural Networks with Application to Regression

TL;DR

This paper establishes theoretical approximation bounds for deep ReLU RNNs in regression tasks, demonstrating minimax optimal prediction error rates under various data assumptions and providing statistical guarantees.

Contribution

It derives the first non-asymptotic approximation bounds for RNNs on H"older functions and applies these to obtain optimal prediction error bounds in regression.

Findings

RNNs can approximate past-dependent H"older functions with controlled error

The derived bounds are minimax optimal under certain data conditions

Provides statistical guarantees for RNN performance in regression

Abstract

We study the approximation capacity of deep ReLU recurrent neural networks (RNNs) and explore the convergence properties of nonparametric least squares regression using RNNs. We derive upper bounds on the approximation error of RNNs for H\"older smooth functions, in the sense that the output at each time step of an RNN can approximate a H\"older function that depends only on past and current information, termed a past-dependent function. This allows a carefully constructed RNN to simultaneously approximate a sequence of past-dependent H\"older functions. We apply these approximation results to derive non-asymptotic upper bounds for the prediction error of the empirical risk minimizer in regression problem. Our error bounds achieve minimax optimal rate under both exponentially $\beta$-mixing and i.i.d. data assumptions, improving upon existing ones. Our results provide statistical guarantees on the performance of RNNs.