Robust Convolution Neural ODEs via Contractivity-promoting regularization

TL;DR

This paper introduces a regularization method based on contraction theory to enhance the robustness of convolutional Neural ODEs against noise and adversarial attacks, demonstrated on image classification benchmarks.

Contribution

It proposes a novel regularization approach to induce contractivity in convolutional Neural ODEs, improving their robustness to input perturbations.

Findings

Contractive Neural ODEs show increased robustness to noise and adversarial attacks.

Regularization via Jacobian and weight constraints effectively promotes contractivity.

Experimental results on MNIST and FashionMNIST validate the approach.

Abstract

Neural networks can be fragile to input noise and adversarial attacks. In this work, we consider Convolutional Neural Ordinary Differential Equations (NODEs), a family of continuous-depth neural networks represented by dynamical systems, and propose to use contraction theory to improve their robustness. For a contractive dynamical system two trajectories starting from different initial conditions converge to each other exponentially fast. Contractive Convolutional NODEs can enjoy increased robustness as slight perturbations of the features do not cause a significant change in the output. Contractivity can be induced during training by using a regularization term involving the Jacobian of the system dynamics. To reduce the computational burden, we show that it can also be promoted using carefully selected weight regularization terms for a class of NODEs with slope-restricted activation functions. The performance of the proposed regularizers is illustrated through benchmark image classification tasks on MNIST and FashionMNIST datasets, where images are corrupted by different kinds of noise and attacks.