A Recipe for Improved Certifiable Robustness

TL;DR

This paper enhances certifiable robustness of neural networks by optimizing Lipschitz-based methods, introducing novel architectural components and data augmentation to significantly improve verification accuracy on benchmarks.

Contribution

The work introduces a comprehensive evaluation and novel design techniques, including Cholesky-orthogonalized residual layers, to advance Lipschitz-based certification methods.

Findings

Significant improvement in deterministic verification accuracy on benchmarks.

Introduction of Cholesky-orthogonalized residual layers enhances network capacity.

Up to 8.5 percentage points increase in verification accuracy achieved.

Abstract

Recent studies have highlighted the potential of Lipschitz-based methods for training certifiably robust neural networks against adversarial attacks. A key challenge, supported both theoretically and empirically, is that robustness demands greater network capacity and more data than standard training. However, effectively adding capacity under stringent Lipschitz constraints has proven more difficult than it may seem, evident by the fact that state-of-the-art approach tend more towards \emph{underfitting} than overfitting. Moreover, we posit that a lack of careful exploration of the design space for Lipshitz-based approaches has left potential performance gains on the table. In this work, we provide a more comprehensive evaluation to better uncover the potential of Lipschitz-based certification methods. Using a combination of novel techniques, design optimizations, and synthesis of prior work, we are able to significantly improve the state-of-the-art VRA for deterministic certification on a variety of benchmark datasets, and over a range of perturbation sizes. Of particular note, we discover that the addition of large ``Cholesky-orthogonalized residual dense'' layers to the end of existing state-of-the-art Lipschitz-controlled ResNet architectures is especially effective for increasing network capacity and performance. Combined with filtered generative data augmentation, our final results further the state of the art deterministic VRA by up to 8.5 percentage points\footnote{Code is available at \url{https://github.com/hukkai/liresnet}}.