Transfer Attacks Revisited: A Large-Scale Empirical Study in Real Computer Vision Settings

TL;DR

This paper presents the first large-scale empirical study of transfer attacks on cloud-based MLaaS platforms, revealing new insights into their properties and vulnerabilities in real-world settings.

Contribution

It provides a comprehensive analysis of transfer attack behaviors in real environments, challenging prior assumptions and uncovering novel properties of transferability.

Findings

Simple surrogates do not always improve transfer attacks.

No single surrogate architecture dominates in real transfer attacks.

Transferability is more influenced by softmax output gaps than logit gaps.

Abstract

One intriguing property of adversarial attacks is their "transferability" -- an adversarial example crafted with respect to one deep neural network (DNN) model is often found effective against other DNNs as well. Intensive research has been conducted on this phenomenon under simplistic controlled conditions. Yet, thus far, there is still a lack of comprehensive understanding about transferability-based attacks ("transfer attacks") in real-world environments. To bridge this critical gap, we conduct the first large-scale systematic empirical study of transfer attacks against major cloud-based MLaaS platforms, taking the components of a real transfer attack into account. The study leads to a number of interesting findings which are inconsistent to the existing ones, including: (1) Simple surrogates do not necessarily improve real transfer attacks. (2) No dominant surrogate architecture is found in real transfer attacks. (3) It is the gap between posterior (output of the softmax layer) rather than the gap between logit (so-called $\kappa$ value) that increases transferability. Moreover, by comparing with prior works, we demonstrate that transfer attacks possess many previously unknown properties in real-world environments, such as (1) Model similarity is not a well-defined concept. (2) $L_2$ norm of perturbation can generate high transferability without usage of gradient and is a more powerful source than $L_\infty$ norm. We believe this work sheds light on the vulnerabilities of popular MLaaS platforms and points to a few promising research directions.