Robustness properties of Facebook's ResNeXt WSL models

TL;DR

This paper evaluates Facebook's ResNeXt WSL models trained on a billion weakly supervised images, demonstrating their exceptional robustness to common corruptions, perturbations, and some adversarial attacks, surpassing previous models in several benchmarks.

Contribution

It provides the first comprehensive analysis of the robustness properties of large-scale weakly supervised ResNeXt models, highlighting their strengths and limitations compared to traditional models.

Findings

ResNeXt WSL models show unprecedented robustness against image corruptions and perturbations.

They achieve state-of-the-art results on ImageNet-C, ImageNet-P, and ImageNet-A benchmarks.

The models exhibit limited adversarial robustness, which declines rapidly with stronger attacks.

Abstract

We investigate the robustness properties of ResNeXt class image recognition models trained with billion scale weakly supervised data (ResNeXt WSL models). These models, recently made public by Facebook AI, were trained with ~1B images from Instagram and fine-tuned on ImageNet. We show that these models display an unprecedented degree of robustness against common image corruptions and perturbations, as measured by the ImageNet-C and ImageNet-P benchmarks. They also achieve substantially improved accuracies on the recently introduced "natural adversarial examples" benchmark (ImageNet-A). The largest of the released models, in particular, achieves state-of-the-art results on ImageNet-C, ImageNet-P, and ImageNet-A by a large margin. The gains on ImageNet-C, ImageNet-P, and ImageNet-A far outpace the gains on ImageNet validation accuracy, suggesting the former as more useful benchmarks to measure further progress in image recognition. Remarkably, the ResNeXt WSL models even achieve a limited degree of adversarial robustness against state-of-the-art white-box attacks (10-step PGD attacks). However, in contrast to adversarially trained models, the robustness of the ResNeXt WSL models rapidly declines with the number of PGD steps, suggesting that these models do not achieve genuine adversarial robustness. Visualization of the learned features also confirms this conclusion. Finally, we show that although the ResNeXt WSL models are more shape-biased than comparable ImageNet-trained models in a shape-texture cue conflict experiment, they still remain much more texture-biased than humans, suggesting that they share some of the underlying characteristics of ImageNet-trained models that make this benchmark challenging.