Dynamical systems' based neural networks

TL;DR

This paper introduces neural networks based on dynamical systems and structure-preserving discretizations, providing theoretical insights and practical robustness, especially for 1-Lipschitz architectures on image datasets.

Contribution

It proposes a novel framework linking neural networks to non-autonomous ODEs, enabling structure incorporation and theoretical analysis of their properties.

Findings

Universal approximation results for the proposed networks

Enhanced robustness of 1-Lipschitz networks against adversarial attacks

Effective application to CIFAR-10 and CIFAR-100 datasets

Abstract

Neural networks have gained much interest because of their effectiveness in many applications. However, their mathematical properties are generally not well understood. If there is some underlying geometric structure inherent to the data or to the function to approximate, it is often desirable to take this into account in the design of the neural network. In this work, we start with a non-autonomous ODE and build neural networks using a suitable, structure-preserving, numerical time-discretisation. The structure of the neural network is then inferred from the properties of the ODE vector field. Besides injecting more structure into the network architectures, this modelling procedure allows a better theoretical understanding of their behaviour. We present two universal approximation results and demonstrate how to impose some particular properties on the neural networks. A particular focus is on 1-Lipschitz architectures including layers that are not 1-Lipschitz. These networks are expressive and robust against adversarial attacks, as shown for the CIFAR-10 and CIFAR-100 datasets.