Structural Extensions of Basis Pursuit: Guarantees on Adversarial Robustness

TL;DR

This paper extends the theoretical guarantees of Basis Pursuit's robustness to adversarial noise to more general network architectures and regularizers, and empirically evaluates shallow sparse coding networks' robustness and efficiency.

Contribution

It generalizes the stability theorem of Basis Pursuit to complex architectures and regularizers, and demonstrates practical robustness and speed advantages in shallow networks.

Findings

Generalized stability guarantees for BP with disjoint group regularization.

Shallow sparse coding networks show robustness against adversarial attacks.

L2 norm-based classification offers accuracy and speed benefits.

Abstract

While deep neural networks are sensitive to adversarial noise, sparse coding using the Basis Pursuit (BP) method is robust against such attacks, including its multi-layer extensions. We prove that the stability theorem of BP holds upon the following generalizations: (i) the regularization procedure can be separated into disjoint groups with different weights, (ii) neurons or full layers may form groups, and (iii) the regularizer takes various generalized forms of the $\ell_1$ norm. This result provides the proof for the architectural generalizations of Cazenavette et al. (2021), including (iv) an approximation of the complete architecture as a shallow sparse coding network. Due to this approximation, we settled to experimenting with shallow networks and studied their robustness against the Iterative Fast Gradient Sign Method on a synthetic dataset and MNIST. We introduce classification based on the $\ell_2$ norms of the groups and show numerically that it can be accurate and offers considerable speedups. In this family, linear transformer shows the best performance. Based on the theoretical results and the numerical simulations, we highlight numerical matters that may improve performance further.