Robust Training and Verification of Implicit Neural Networks: A Non-Euclidean Contractive Approach

TL;DR

This paper introduces a non-Euclidean contraction theory-based framework for training and verifying the robustness of implicit neural networks, using reachability analysis and Lipschitz bounds to improve robustness guarantees.

Contribution

It develops a novel theoretical and computational approach leveraging non-Euclidean contraction theory for implicit neural network training and robustness verification.

Findings

The proposed methods effectively compute Lipschitz constants and reachability bounds.

Algorithms demonstrate improved robustness verification on MNIST.

The framework outperforms existing approaches in robustness assessment.

Abstract

This paper proposes a theoretical and computational framework for training and robustness verification of implicit neural networks based upon non-Euclidean contraction theory. The basic idea is to cast the robustness analysis of a neural network as a reachability problem and use (i) the $\ell_{\infty}$-norm input-output Lipschitz constant and (ii) the tight inclusion function of the network to over-approximate its reachable sets. First, for a given implicit neural network, we use $\ell_{\infty}$-matrix measures to propose sufficient conditions for its well-posedness, design an iterative algorithm to compute its fixed points, and provide upper bounds for its $\ell_\infty$-norm input-output Lipschitz constant. Second, we introduce a related embedded network and show that the embedded network can be used to provide an $\ell_\infty$-norm box over-approximation of the reachable sets of the original network. Moreover, we use the embedded network to design an iterative algorithm for computing the upper bounds of the original system's tight inclusion function. Third, we use the upper bounds of the Lipschitz constants and the upper bounds of the tight inclusion functions to design two algorithms for the training and robustness verification of implicit neural networks. Finally, we apply our algorithms to train implicit neural networks on the MNIST dataset and compare the robustness of our models with the models trained via existing approaches in the literature.