Standalone Neural ODEs with Sensitivity Analysis

TL;DR

This paper introduces the standalone Neural ODE (sNODE) model with a novel training scheme and sensitivity analysis, enhancing robustness, explainability, and uncertainty quantification in deep neural networks.

Contribution

The paper proposes sNODE, a continuous-depth neural network with a new NCG training method incorporating Sobolev gradients, and introduces a sensitivity analysis framework for robustness and explainability.

Findings

sNODE outperforms ResNet in robustness and accuracy

Sensitivity analysis effectively measures uncertainty propagation

Method enhances model explainability and adversarial robustness

Abstract

This paper presents the Standalone Neural ODE (sNODE), a continuous-depth neural ODE model capable of describing a full deep neural network. This uses a novel nonlinear conjugate gradient (NCG) descent optimization scheme for training, where the Sobolev gradient can be incorporated to improve smoothness of model weights. We also present a general formulation of the neural sensitivity problem and show how it is used in the NCG training. The sensitivity analysis provides a reliable measure of uncertainty propagation throughout a network, and can be used to study model robustness and to generate adversarial attacks. Our evaluations demonstrate that our novel formulations lead to increased robustness and performance as compared to ResNet models, and that it opens up for new opportunities for designing and developing machine learning with improved explainability.