From NeurODEs to AutoencODEs: a mean-field control framework for width-varying Neural Networks

TL;DR

This paper extends the NeurODE framework to variable-width neural networks by introducing AutoencODEs, enabling continuous-time modeling of more complex architectures with practical training methods validated through experiments.

Contribution

It proposes AutoencODEs, a novel continuous-time control framework for width-varying neural networks, expanding the applicability of mean-field control to more realistic deep learning models.

Findings

AutoencODEs successfully model variable-width neural networks.

Theoretical analysis under low Tikhonov regularization reveals local convexity regions.

Numerical experiments demonstrate the effectiveness of the proposed training method.

Abstract

The connection between Residual Neural Networks (ResNets) and continuous-time control systems (known as NeurODEs) has led to a mathematical analysis of neural networks which has provided interesting results of both theoretical and practical significance. However, by construction, NeurODEs have been limited to describing constant-width layers, making them unsuitable for modeling deep learning architectures with layers of variable width. In this paper, we propose a continuous-time Autoencoder, which we call AutoencODE, based on a modification of the controlled field that drives the dynamics. This adaptation enables the extension of the mean-field control framework originally devised for conventional NeurODEs. In this setting, we tackle the case of low Tikhonov regularization, resulting in potentially non-convex cost landscapes. While the global results obtained for high Tikhonov regularization may not hold globally, we show that many of them can be recovered in regions where the loss function is locally convex. Inspired by our theoretical findings, we develop a training method tailored to this specific type of Autoencoders with residual connections, and we validate our approach through numerical experiments conducted on various examples.