Back in Black: A Comparative Evaluation of Recent State-Of-The-Art Black-Box Attacks

TL;DR

This paper systematically reviews and categorizes recent black-box adversarial attacks in machine learning, providing a new framework for fair comparison and insights into attack success rates across different models.

Contribution

It offers a comprehensive survey of 20 recent black-box attacks, introduces a novel mathematical framework for comparison, and analyzes success rates based on attack models.

Findings

Categorized attacks into four main types.

Provided a new framework for fair comparison.

Analyzed success rates relative to attack models.

Abstract

The field of adversarial machine learning has experienced a near exponential growth in the amount of papers being produced since 2018. This massive information output has yet to be properly processed and categorized. In this paper, we seek to help alleviate this problem by systematizing the recent advances in adversarial machine learning black-box attacks since 2019. Our survey summarizes and categorizes 20 recent black-box attacks. We also present a new analysis for understanding the attack success rate with respect to the adversarial model used in each paper. Overall, our paper surveys a wide body of literature to highlight recent attack developments and organizes them into four attack categories: score based attacks, decision based attacks, transfer attacks and non-traditional attacks. Further, we provide a new mathematical framework to show exactly how attack results can fairly be compared.