Improved techniques for deterministic l2 robustness

TL;DR

This paper introduces new techniques to improve the training efficiency and robustness of 1-Lipschitz CNNs under the l2 norm, achieving state-of-the-art results on CIFAR datasets.

Contribution

It proposes a robustness certification method, reduces training time for SOC layers, and introduces new pooling layers, advancing the performance of provably robust CNNs.

Findings

Achieved +1.79% and +3.82% standard and robust accuracy on CIFAR-10.

Achieved +3.78% and +4.75% on CIFAR-100.

Reduced training time for SOC layers by over 30%.

Abstract

Training convolutional neural networks (CNNs) with a strict 1-Lipschitz constraint under the $l_{2}$ norm is useful for adversarial robustness, interpretable gradients and stable training. 1-Lipschitz CNNs are usually designed by enforcing each layer to have an orthogonal Jacobian matrix (for all inputs) to prevent the gradients from vanishing during backpropagation. However, their performance often significantly lags behind that of heuristic methods to enforce Lipschitz constraints where the resulting CNN is not \textit{provably} 1-Lipschitz. In this work, we reduce this gap by introducing (a) a procedure to certify robustness of 1-Lipschitz CNNs by replacing the last linear layer with a 1-hidden layer MLP that significantly improves their performance for both standard and provably robust accuracy, (b) a method to significantly reduce the training time per epoch for Skew Orthogonal Convolution (SOC) layers (>30\% reduction for deeper networks) and (c) a class of pooling layers using the mathematical property that the $l_{2}$ distance of an input to a manifold is 1-Lipschitz. Using these methods, we significantly advance the state-of-the-art for standard and provable robust accuracies on CIFAR-10 (gains of +1.79\% and +3.82\%) and similarly on CIFAR-100 (+3.78\% and +4.75\%) across all networks. Code is available at \url{https://github.com/singlasahil14/improved_l2_robustness}.