Interpolation and approximation via Momentum ResNets and Neural ODEs

TL;DR

This paper investigates how memory terms in continuous-layer neural networks like Momentum ResNets and Neural ODEs affect their ability to interpolate and approximate complex functions, supported by theoretical analysis and simulations.

Contribution

It introduces and analyzes two models with memory: Momentum ResNets and Neural ODEs with auxiliary states, highlighting their approximation capabilities.

Findings

Both models exhibit universal approximation properties.

Neural ODEs with auxiliary states can represent time-dependent functions.

Numerical simulations validate theoretical results.

Abstract

In this article, we explore the effects of memory terms in continuous-layer Deep Residual Networks by studying Neural ODEs (NODEs). We investigate two types of models. On one side, we consider the case of Residual Neural Networks with dependence on multiple layers, more precisely Momentum ResNets. On the other side, we analyze a Neural ODE with auxiliary states playing the role of memory states. We examine the interpolation and universal approximation properties for both architectures through a simultaneous control perspective. We also prove the ability of the second model to represent sophisticated maps, such as parametrizations of time-dependent functions. Numerical simulations complement our study.