Continuous-in-Depth Neural Networks

TL;DR

ContinuousNet introduces a continuous-in-depth neural network architecture inspired by advanced numerical integration schemes, enabling flexible evaluation, efficient training, and inference with minimal accuracy loss.

Contribution

It proposes ContinuousNet, a novel continuous-in-depth neural network model that leverages higher-order numerical schemes for improved flexibility and efficiency.

Findings

ContinuousNets can be evaluated with different step sizes and schemes.

Incremental-in-depth training improves model quality and reduces training time.

Decreasing units in the graph allows faster inference with little accuracy loss.

Abstract

Recent work has attempted to interpret residual networks (ResNets) as one step of a forward Euler discretization of an ordinary differential equation, focusing mainly on syntactic algebraic similarities between the two systems. Discrete dynamical integrators of continuous dynamical systems, however, have a much richer structure. We first show that ResNets fail to be meaningful dynamical integrators in this richer sense. We then demonstrate that neural network models can learn to represent continuous dynamical systems, with this richer structure and properties, by embedding them into higher-order numerical integration schemes, such as the Runge Kutta schemes. Based on these insights, we introduce ContinuousNet as a continuous-in-depth generalization of ResNet architectures. ContinuousNets exhibit an invariance to the particular computational graph manifestation. That is, the continuous-in-depth model can be evaluated with different discrete time step sizes, which changes the number of layers, and different numerical integration schemes, which changes the graph connectivity. We show that this can be used to develop an incremental-in-depth training scheme that improves model quality, while significantly decreasing training time. We also show that, once trained, the number of units in the computational graph can even be decreased, for faster inference with little-to-no accuracy drop.