Continuous limits of residual neural networks in case of large input data

TL;DR

This paper explores the mathematical limits of residual neural networks with large input data, deriving mean-field descriptions and analyzing training dynamics through optimal control, supported by numerical experiments.

Contribution

It introduces a mean-field limit for ResNets with large input data and studies the training process using controllability and optimal control frameworks.

Findings

Derived a mean-field limit for large-scale ResNets

Proved well-posedness of the neural differential equations

Numerical simulations support theoretical results

Abstract

Residual deep neural networks (ResNets) are mathematically described as interacting particle systems. In the case of infinitely many layers the ResNet leads to a system of coupled system of ordinary differential equations known as neural differential equations. For large scale input data we derive a mean--field limit and show well--posedness of the resulting description. Further, we analyze the existence of solutions to the training process by using both a controllability and an optimal control point of view. Numerical investigations based on the solution of a formal optimality system illustrate the theoretical findings.