State Space Representations of Deep Neural Networks

TL;DR

This paper models neural networks with skip connections as dynamical systems, deriving closed-form state space representations, and shows that higher-order smoothness increases the effective embedding dimension, reducing parameter requirements.

Contribution

It introduces a novel dynamical systems perspective for neural networks with skip connections and derives explicit state space solutions for higher-order smooth networks.

Findings

Higher-order smoothness increases the state space dimension proportionally.

Networks with higher-order smoothness require fewer parameters for the same embedding.

Numerical simulations validate the theoretical state space representations.

Abstract

This paper deals with neural networks as dynamical systems governed by differential or difference equations. It shows that the introduction of skip connections into network architectures, such as residual networks and dense networks, turns a system of static equations into a system of dynamical equations with varying levels of smoothness on the layer-wise transformations. Closed form solutions for the state space representations of general dense networks, as well as $k^{th}$ order smooth networks, are found in general settings. Furthermore, it is shown that imposing $k^{th}$ order smoothness on a network architecture with $d$-many nodes per layer increases the state space dimension by a multiple of $k$, and so the effective embedding dimension of the data manifold is $k \cdot d$-many dimensions. It follows that network architectures of these types reduce the number of parameters needed to maintain the same embedding dimension by a factor of $k^2$ when compared to an equivalent first-order, residual network, significantly motivating the development of network architectures of these types. Numerical simulations were run to validate parts of the developed theory.