Universal Approximation Properties for an ODENet and a ResNet: Mathematical Analysis and Numerical Experiments

TL;DR

This paper establishes the universal approximation capabilities of ODENet and ResNet models with skip connections, providing theoretical proofs and numerical experiments demonstrating their effectiveness in approximating functions and learning tasks.

Contribution

It proves the universal approximation property for ODENet and ResNet models, including explicit gradient derivation and a practical learning algorithm.

Findings

ODENet of width n+m can approximate any continuous function on compact sets.

ResNet has the same approximation property as depth tends to infinity.

The proposed learning algorithm performs well on MNIST classification tasks.

Abstract

We prove a universal approximation property (UAP) for a class of ODENet and a class of ResNet, which are simplified mathematical models for deep learning systems with skip connections. The UAP can be stated as follows. Let $n$ and $m$ be the dimension of input and output data, and assume $m\leq n$. Then we show that ODENet of width $n+m$ with any non-polynomial continuous activation function can approximate any continuous function on a compact subset on $\mathbb{R}^n$. We also show that ResNet has the same property as the depth tends to infinity. Furthermore, we derive the gradient of a loss function explicitly with respect to a certain tuning variable. We use this to construct a learning algorithm for ODENet. To demonstrate the usefulness of this algorithm, we apply it to a regression problem, a binary classification, and a multinomial classification in MNIST.