Deep learning and the rate of approximation by flows

TL;DR

This paper explores how the depth of deep residual networks influences their approximation capabilities, framing the problem in a continuous dynamical systems context and linking it to geometric properties of diffeomorphisms.

Contribution

It introduces a geometric framework using sub-Finsler manifolds to quantify the minimal time for approximation, connecting learning efficiency to architectural compatibility.

Findings

Approximation capacity relates to geodesic distances on diffeomorphism manifolds.

Deep learning approximation differs fundamentally from linear approximation theory.

The minimal time for approximation is characterized by a variational principle.

Abstract

We investigate the dependence of the approximation capacity of deep residual networks on its depth in a continuous dynamical systems setting. This can be formulated as the general problem of quantifying the minimal time-horizon required to approximate a diffeomorphism by flows driven by a given family $\mathcal F$ of vector fields. We show that this minimal time can be identified as a geodesic distance on a sub-Finsler manifold of diffeomorphisms, where the local geometry is characterised by a variational principle involving $\mathcal F$. This connects the learning efficiency of target relationships to their compatibility with the learning architectural choice. Further, the results suggest that the key approximation mechanism in deep learning, namely the approximation of functions by composition or dynamics, differs in a fundamental way from linear approximation theory, where linear spaces and norm-based rate estimates are replaced by manifolds and geodesic distances.