Universal approximation property of ODENet and ResNet with a single activation function

TL;DR

This paper proves that ODENet and ResNet models with a single activation function can universally approximate more complex models, demonstrating their theoretical expressive power in deep learning.

Contribution

It establishes the universal approximation property of ODENet and ResNet with a single activation function, extending understanding of their capabilities.

Findings

ODENet and ResNet can approximate any vector field with a single activation function.

The approximation is uniform over finite intervals.

The results apply to common activation functions used in practice.

Abstract

We study a universal approximation property of ODENet and ResNet. The ODENet is a map from an initial value to the final value of an ODE system in a finite interval. It is considered a mathematical model of a ResNet-type deep learning system. We consider dynamical systems with vector fields given by a single composition of the activation function and an affine mapping, which is the most common choice of the ODENet or ResNet vector field in actual machine learning systems. We show that such an ODENet and ResNet with a restricted vector field can uniformly approximate ODENet with a general vector field.