Metric-Entropy Limits on the Approximation of Nonlinear Dynamical Systems

TL;DR

This paper establishes fundamental limits on approximating nonlinear dynamical systems using recurrent neural networks, showing RNNs can optimally approximate systems with certain Lipschitz properties in terms of metric entropy.

Contribution

It provides a refined metric-entropy analysis for classes of nonlinear systems and demonstrates RNNs achieve these approximation limits, advancing understanding of neural network capabilities.

Findings

RNNs can approximate Lipschitz fading-memory systems optimally.

Metric-entropy characterization involves order, type, and generalized dimension.

RNN approximation matches the fundamental limits for certain nonlinear systems.

Abstract

This paper is concerned with fundamental limits on the approximation of nonlinear dynamical systems. Specifically, we show that recurrent neural networks (RNNs) can approximate nonlinear systems -- that satisfy a Lipschitz property and forget past inputs fast enough -- in metric-entropy-optimal manner. As the sets of sequence-to-sequence mappings realized by the dynamical systems we consider are significantly more massive than function classes generally analyzed in approximation theory, a refined metric-entropy characterization is needed, namely in terms of order, type, and generalized dimension. We compute these quantities for the classes of exponentially- and polynomially Lipschitz fading-memory systems and show that RNNs can achieve them.