Simpler Certified Radius Maximization by Propagating Covariances

TL;DR

This paper introduces a method to directly propagate covariances in neural networks to efficiently maximize the certified radius for adversarial robustness, reducing sampling complexity with minor accuracy trade-offs.

Contribution

It proposes a novel covariance propagation technique for certified radius maximization, improving efficiency and scalability over traditional Monte Carlo sampling methods.

Findings

Effective on datasets like Cifar-10, ImageNet, and Places365.

Offers runtime savings with moderate network depth.

Achieves a small accuracy compromise for increased efficiency.

Abstract

One strategy for adversarially training a robust model is to maximize its certified radius -- the neighborhood around a given training sample for which the model's prediction remains unchanged. The scheme typically involves analyzing a "smoothed" classifier where one estimates the prediction corresponding to Gaussian samples in the neighborhood of each sample in the mini-batch, accomplished in practice by Monte Carlo sampling. In this paper, we investigate the hypothesis that this sampling bottleneck can potentially be mitigated by identifying ways to directly propagate the covariance matrix of the smoothed distribution through the network. To this end, we find that other than certain adjustments to the network, propagating the covariances must also be accompanied by additional accounting that keeps track of how the distributional moments transform and interact at each stage in the network. We show how satisfying these criteria yields an algorithm for maximizing the certified radius on datasets including Cifar-10, ImageNet, and Places365 while offering runtime savings on networks with moderate depth, with a small compromise in overall accuracy. We describe the details of the key modifications that enable practical use. Via various experiments, we evaluate when our simplifications are sensible, and what the key benefits and limitations are.