Compositional Curvature Bounds for Deep Neural Networks

TL;DR

This paper introduces a scalable method to compute and control the second derivative bounds of deep neural networks, improving their robustness against adversarial attacks by leveraging compositional structure.

Contribution

We develop a novel layer-wise algorithm to compute curvature bounds, enabling regularization of neural networks for enhanced adversarial robustness.

Findings

Effective curvature bounds improve robustness against attacks.

Method scales to large networks and datasets.

Regularization based on bounds enhances model safety.

Abstract

A key challenge that threatens the widespread use of neural networks in safety-critical applications is their vulnerability to adversarial attacks. In this paper, we study the second-order behavior of continuously differentiable deep neural networks, focusing on robustness against adversarial perturbations. First, we provide a theoretical analysis of robustness and attack certificates for deep classifiers by leveraging local gradients and upper bounds on the second derivative (curvature constant). Next, we introduce a novel algorithm to analytically compute provable upper bounds on the second derivative of neural networks. This algorithm leverages the compositional structure of the model to propagate the curvature bound layer-by-layer, giving rise to a scalable and modular approach. The proposed bound can serve as a differentiable regularizer to control the curvature of neural networks during training, thereby enhancing robustness. Finally, we demonstrate the efficacy of our method on classification tasks using the MNIST and CIFAR-10 datasets.