Rethinking Robust Contrastive Learning from the Adversarial Perspective

TL;DR

This paper investigates how adversarial training influences contrastive learning, revealing that it reduces disparities between adversarial and clean representations and promotes convergence towards a universal representation, thereby enhancing robustness.

Contribution

It provides a comprehensive analysis of adversarial training effects on contrastive learning, highlighting its role in aligning representations and improving network robustness.

Findings

Adversarial training reduces disparities between adversarial and clean representations.

It promotes convergence of representations toward a universal set.

Increasing similarity between adversarial and clean representations enhances robustness.

Abstract

To advance the understanding of robust deep learning, we delve into the effects of adversarial training on self-supervised and supervised contrastive learning alongside supervised learning. Our analysis uncovers significant disparities between adversarial and clean representations in standard-trained networks across various learning algorithms. Remarkably, adversarial training mitigates these disparities and fosters the convergence of representations toward a universal set, regardless of the learning scheme used. Additionally, increasing the similarity between adversarial and clean representations, particularly near the end of the network, enhances network robustness. These findings offer valuable insights for designing and training effective and robust deep learning networks. Our code is released at \textcolor{magenta}{\url{https://github.com/softsys4ai/CL-Robustness}}.