Interpolation, Approximation and Controllability of Deep Neural Networks

TL;DR

This paper explores the expressive capabilities of deep residual neural networks modeled as control systems, focusing on their ability to interpolate and approximate functions, and clarifies the conditions under which these properties are equivalent or distinct.

Contribution

It provides a theoretical characterization of universal interpolation and approximation in neural networks viewed as control systems, highlighting their relationship and differences.

Findings

Universal interpolation holds for almost any architecture with non-linearity.

Universal interpolation and approximation are generally not deducible from each other.

Conditions are identified under which interpolation and approximation are equivalent.

Abstract

We investigate the expressive power of deep residual neural networks idealized as continuous dynamical systems through control theory. Specifically, we consider two properties that arise from supervised learning, namely universal interpolation - the ability to match arbitrary input and target training samples - and the closely related notion of universal approximation - the ability to approximate input-target functional relationships via flow maps. Under the assumption of affine invariance of the control family, we give a characterisation of universal interpolation, showing that it holds for essentially any architecture with non-linearity. Furthermore, we elucidate the relationship between universal interpolation and universal approximation in the context of general control systems, showing that the two properties cannot be deduced from each other. At the same time, we identify conditions on the control family and the target function that ensures the equivalence of the two notions.