Robust Pre-Training by Adversarial Contrastive Learning

TL;DR

This paper introduces Adversarial Contrastive Learning (ACL), a pre-training method that enhances robustness and label efficiency by learning feature-invariant representations under data augmentations and adversarial perturbations.

Contribution

It proposes a novel contrastive learning framework that incorporates adversarial perturbations to improve robustness and efficiency of self-supervised pre-training.

Findings

ACL outperforms previous unsupervised robust pre-training methods by nearly 3% on CIFAR-10.

ACL improves both robust and standard accuracy, demonstrating enhanced feature invariance.

Pre-training with ACL benefits semi-supervised adversarial training with limited labeled data.

Abstract

Recent work has shown that, when integrated with adversarial training, self-supervised pre-training can lead to state-of-the-art robustness In this work, we improve robustness-aware self-supervised pre-training by learning representations that are consistent under both data augmentations and adversarial perturbations. Our approach leverages a recent contrastive learning framework, which learns representations by maximizing feature consistency under differently augmented views. This fits particularly well with the goal of adversarial robustness, as one cause of adversarial fragility is the lack of feature invariance, i.e., small input perturbations can result in undesirable large changes in features or even predicted labels. We explore various options to formulate the contrastive task, and demonstrate that by injecting adversarial perturbations, contrastive pre-training can lead to models that are both label-efficient and robust. We empirically evaluate the proposed Adversarial Contrastive Learning (ACL) and show it can consistently outperform existing methods. For example on the CIFAR-10 dataset, ACL outperforms the previous state-of-the-art unsupervised robust pre-training approach by 2.99% on robust accuracy and 2.14% on standard accuracy. We further demonstrate that ACL pre-training can improve semi-supervised adversarial training, even when only a few labeled examples are available. Our codes and pre-trained models have been released at: https://github.com/VITA-Group/Adversarial-Contrastive-Learning.