BinarySelect to Improve Accessibility of Black-Box Attack Research

TL;DR

This paper introduces BinarySelect, a method that significantly reduces the number of queries needed for black-box adversarial text attacks, making such research more accessible and resource-efficient.

Contribution

BinarySelect combines binary search with attack selection to efficiently identify tokens for adversarial attacks, reducing query complexity from linear to logarithmic scale.

Findings

BinarySelect reduces query count by approximately 32% on Yelp dataset.

The attack effectiveness drops only marginally (around 5 points) despite fewer queries.

BinarySelect enables resource-efficient adversarial attack research on NLP models.

Abstract

Adversarial text attack research is useful for testing the robustness of NLP models, however, the rise of transformers has greatly increased the time required to test attacks. Especially when researchers do not have access to adequate resources (e.g. GPUs). This can hinder attack research, as modifying one example for an attack can require hundreds of queries to a model, especially for black-box attacks. Often these attacks remove one token at a time to find the ideal one to change, requiring $n$ queries (the length of the text) right away. We propose a more efficient selection method called BinarySelect which combines binary search and attack selection methods to greatly reduce the number of queries needed to find a token. We find that BinarySelect only needs $\text{log}_2(n) * 2$ queries to find the first token compared to $n$ queries. We also test BinarySelect in an attack setting against 5 classifiers across 3 datasets and find a viable tradeoff between number of queries saved and attack effectiveness. For example, on the Yelp dataset, the number of queries is reduced by 32% (72 less) with a drop in attack effectiveness of only 5 points. We believe that BinarySelect can help future researchers study adversarial attacks and black-box problems more efficiently and opens the door for researchers with access to less resources.