Designing Stable Neural Networks using Convex Analysis and ODEs

TL;DR

This paper introduces a neural network architecture inspired by ODEs and convex analysis that maintains non-expansiveness, ensuring stability and robustness, and demonstrates its effectiveness in image classification, denoising, and deblurring tasks.

Contribution

The paper proposes a novel ResNet-inspired architecture that encodes non-expansive operators using spectral norm constraints, ensuring stability and robustness in deep networks.

Findings

The architecture maintains non-expansiveness with spectral norm constraints.

It achieves competitive performance in adversarial robustness and image restoration.

The method allows effective training despite spectral norm constraints.

Abstract

Motivated by classical work on the numerical integration of ordinary differential equations we present a ResNet-styled neural network architecture that encodes non-expansive (1-Lipschitz) operators, as long as the spectral norms of the weights are appropriately constrained. This is to be contrasted with the ordinary ResNet architecture which, even if the spectral norms of the weights are constrained, has a Lipschitz constant that, in the worst case, grows exponentially with the depth of the network. Further analysis of the proposed architecture shows that the spectral norms of the weights can be further constrained to ensure that the network is an averaged operator, making it a natural candidate for a learned denoiser in Plug-and-Play algorithms. Using a novel adaptive way of enforcing the spectral norm constraints, we show that, even with these constraints, it is possible to train performant networks. The proposed architecture is applied to the problem of adversarially robust image classification, to image denoising, and finally to the inverse problem of deblurring.