Contrastive Self-Supervised Learning Leads to Higher Adversarial Susceptibility

TL;DR

This paper investigates why contrastive self-supervised learning (CSL) is more susceptible to adversarial attacks than supervised learning, attributing it to false negative pairs and proposing a method to improve robustness.

Contribution

It identifies false negative pairs as a key factor in CSL's vulnerability and introduces a simple strategy to enhance robustness, closing the adversarial robustness gap.

Findings

CSL has higher sensitivity to adversarial perturbations than supervised learning.

Removing false negative pairs improves CSL robustness significantly.

The proposed method increases robustness by up to 68% in experiments.

Abstract

Contrastive self-supervised learning (CSL) has managed to match or surpass the performance of supervised learning in image and video classification. However, it is still largely unknown if the nature of the representations induced by the two learning paradigms is similar. We investigate this under the lens of adversarial robustness. Our analysis of the problem reveals that CSL has intrinsically higher sensitivity to perturbations over supervised learning. We identify the uniform distribution of data representation over a unit hypersphere in the CSL representation space as the key contributor to this phenomenon. We establish that this is a result of the presence of false negative pairs in the training process, which increases model sensitivity to input perturbations. Our finding is supported by extensive experiments for image and video classification using adversarial perturbations and other input corruptions. We devise a strategy to detect and remove false negative pairs that is simple, yet effective in improving model robustness with CSL training. We close up to 68% of the robustness gap between CSL and its supervised counterpart. Finally, we contribute to adversarial learning by incorporating our method in CSL. We demonstrate an average gain of about 5% over two different state-of-the-art methods in this domain.