RUSH: Robust Contrastive Learning via Randomized Smoothing

TL;DR

This paper introduces RUSH, a robust contrastive learning method that combines contrastive pre-training with randomized smoothing, achieving high robustness and accuracy with lower training costs.

Contribution

It reveals an implicit robustness in contrastive pre-training and leverages it to develop RUSH, a new algorithm that enhances robustness without expensive adversarial training.

Findings

RUSH outperforms adversarial training-based robust classifiers on benchmarks.

RUSH achieves 77.8% robust accuracy under PGD attack on CIFAR-10.

RUSH significantly reduces training costs compared to traditional adversarial training.

Abstract

Recently, adversarial training has been incorporated in self-supervised contrastive pre-training to augment label efficiency with exciting adversarial robustness. However, the robustness came at a cost of expensive adversarial training. In this paper, we show a surprising fact that contrastive pre-training has an interesting yet implicit connection with robustness, and such natural robustness in the pre trained representation enables us to design a powerful robust algorithm against adversarial attacks, RUSH, that combines the standard contrastive pre-training and randomized smoothing. It boosts both standard accuracy and robust accuracy, and significantly reduces training costs as compared with adversarial training. We use extensive empirical studies to show that the proposed RUSH outperforms robust classifiers from adversarial training, by a significant margin on common benchmarks (CIFAR-10, CIFAR-100, and STL-10) under first-order attacks. In particular, under $\ell_{\infty}$-norm perturbations of size 8/255 PGD attack on CIFAR-10, our model using ResNet-18 as backbone reached 77.8% robust accuracy and 87.9% standard accuracy. Our work has an improvement of over 15% in robust accuracy and a slight improvement in standard accuracy, compared to the state-of-the-arts.