Benchmarking Robust Self-Supervised Learning Across Diverse Downstream Tasks

TL;DR

This paper evaluates the robustness of self-supervised vision models across various downstream tasks, revealing vulnerabilities and the need for broader robustness improvements beyond image classification.

Contribution

It provides the first comprehensive empirical analysis of adversarial robustness of self-supervised vision encoders across multiple tasks.

Findings

State-of-the-art adversarial fine-tuning degrades performance on non-classification tasks.

Current robustness techniques are insufficient for diverse downstream applications.

Robustness improvements need to be generalized beyond image classification.

Abstract

Large-scale vision models have become integral in many applications due to their unprecedented performance and versatility across downstream tasks. However, the robustness of these foundation models has primarily been explored for a single task, namely image classification. The vulnerability of other common vision tasks, such as semantic segmentation and depth estimation, remains largely unknown. We present a comprehensive empirical evaluation of the adversarial robustness of self-supervised vision encoders across multiple downstream tasks. Our attacks operate in the encoder embedding space and at the downstream task output level. In both cases, current state-of-the-art adversarial fine-tuning techniques tested only for classification significantly degrade clean and robust performance on other tasks. Since the purpose of a foundation model is to cater to multiple applications at once, our findings reveal the need to enhance encoder robustness more broadly. Our code is available at ${github.com/layer6ai-labs/ssl-robustness}$.