When Does Contrastive Learning Preserve Adversarial Robustness from Pretraining to Finetuning?

TL;DR

This paper investigates how contrastive learning can be adapted to better preserve adversarial robustness during the transition from pretraining to finetuning, introducing new methods to enhance robustness transfer.

Contribution

It proposes AdvCL, a novel adversarial contrastive learning framework that improves robustness transfer without sacrificing accuracy or finetuning efficiency.

Findings

AdvCL outperforms existing methods on multiple datasets.

High-frequency image components improve model robustness.

Pseudo-supervision helps preserve robustness during finetuning.

Abstract

Contrastive learning (CL) can learn generalizable feature representations and achieve the state-of-the-art performance of downstream tasks by finetuning a linear classifier on top of it. However, as adversarial robustness becomes vital in image classification, it remains unclear whether or not CL is able to preserve robustness to downstream tasks. The main challenge is that in the self-supervised pretraining + supervised finetuning paradigm, adversarial robustness is easily forgotten due to a learning task mismatch from pretraining to finetuning. We call such a challenge 'cross-task robustness transferability'. To address the above problem, in this paper we revisit and advance CL principles through the lens of robustness enhancement. We show that (1) the design of contrastive views matters: High-frequency components of images are beneficial to improving model robustness; (2) Augmenting CL with pseudo-supervision stimulus (e.g., resorting to feature clustering) helps preserve robustness without forgetting. Equipped with our new designs, we propose AdvCL, a novel adversarial contrastive pretraining framework. We show that AdvCL is able to enhance cross-task robustness transferability without loss of model accuracy and finetuning efficiency. With a thorough experimental study, we demonstrate that AdvCL outperforms the state-of-the-art self-supervised robust learning methods across multiple datasets (CIFAR-10, CIFAR-100, and STL-10) and finetuning schemes (linear evaluation and full model finetuning).