Approximation Capabilities of Neural ODEs and Invertible Residual Networks

TL;DR

This paper investigates the approximation limits of Neural ODEs and i-ResNets, showing their constraints and how they can be extended to approximate all continuous functions with simple modifications.

Contribution

It proves the limitations of Neural ODEs and i-ResNets in modeling invertible functions and demonstrates how to achieve universal approximation with minimal modifications.

Findings

Neural ODEs and i-ResNets are limited in their invertible function approximation.

Any homeomorphism can be approximated by Neural ODEs and i-ResNets in higher dimensions.

Adding a linear layer makes these models universal approximators for all continuous functions.

Abstract

Neural ODEs and i-ResNet are recently proposed methods for enforcing invertibility of residual neural models. Having a generic technique for constructing invertible models can open new avenues for advances in learning systems, but so far the question of whether Neural ODEs and i-ResNets can model any continuous invertible function remained unresolved. Here, we show that both of these models are limited in their approximation capabilities. We then prove that any homeomorphism on a $p$-dimensional Euclidean space can be approximated by a Neural ODE operating on a $2p$-dimensional Euclidean space, and a similar result for i-ResNets. We conclude by showing that capping a Neural ODE or an i-ResNet with a single linear layer is sufficient to turn the model into a universal approximator for non-invertible continuous functions.