Imbedding Deep Neural Networks

TL;DR

This paper introduces a novel approach to neural network depth by using invariant imbedding, transforming the problem into forward initial value problems, and demonstrating competitive performance in supervised learning and time series tasks.

Contribution

It proposes a new neural network architecture based on invariant imbedding, providing a theoretical framework and practical tools for analyzing and implementing network depth.

Findings

The new architectures are competitive in supervised learning.

The method offers a theoretical insight into network depth properties.

Experimental results show effectiveness in time series prediction.

Abstract

Continuous-depth neural networks, such as Neural ODEs, have refashioned the understanding of residual neural networks in terms of non-linear vector-valued optimal control problems. The common solution is to use the adjoint sensitivity method to replicate a forward-backward pass optimisation problem. We propose a new approach which explicates the network's `depth' as a fundamental variable, thus reducing the problem to a system of forward-facing initial value problems. This new method is based on the principle of `Invariant Imbedding' for which we prove a general solution, applicable to all non-linear, vector-valued optimal control problems with both running and terminal loss. Our new architectures provide a tangible tool for inspecting the theoretical--and to a great extent unexplained--properties of network depth. They also constitute a resource of discrete implementations of Neural ODEs comparable to classes of imbedded residual neural networks. Through a series of experiments, we show the competitive performance of the proposed architectures for supervised learning and time series prediction.