Upper Approximation Bounds for Neural Oscillators

TL;DR

This paper establishes theoretical upper bounds on the approximation capabilities of neural oscillators, demonstrating their efficiency in modeling complex dynamical systems with polynomial error scaling.

Contribution

It derives novel approximation bounds for neural oscillators, extending the theory to state-space models and validating results with numerical experiments.

Findings

Approximation error scales polynomially with MLP widths.

Bounds apply to causal, continuous operators and stable second-order systems.

Numerical experiments confirm theoretical convergence rates.

Abstract

Neural oscillators, originating from second-order ordinary differential equations (ODEs), have demonstrated strong performance in stably learning causal mappings between long-term sequences or continuous temporal functions, as well as in accurately approximating physical systems. However, theoretically quantifying the capacities of their neural network architectures remains a significant challenge. In this study, the neural oscillator consisting of a second-order ODE followed by a multilayer perceptron (MLP) is considered. Its upper approximation bound for approximating causal and uniformly continuous operators between continuous temporal function spaces and that for approximating uniformly asymptotically incrementally stable second-order dynamical systems are derived. The established proof method of the approximation bound for approximating the causal continuous operators can also be directly applied to state-space models consisting of a linear time-continuous complex recurrent neural network followed by an MLP. Theoretical results reveal that the approximation error of the neural oscillator for approximating the second-order dynamical systems scales polynomially with the reciprocals of the widths of two utilized MLPs, thus overcoming the curse of parametric complexity. The convergence rates of two established approximation error bounds are validated through four numerical cases. These results provide a robust theoretical foundation for the effective application of the neural oscillator in science and engineering.