Hard-label Manifolds: Unexpected Advantages of Query Efficiency for Finding On-manifold Adversarial Examples

TL;DR

This paper explores how query efficiency in hard-label adversarial attacks relates to traversing the data manifold, revealing that such attacks can produce more realistic adversarial examples and informing future robust model design.

Contribution

It introduces an information-theoretic framework linking query efficiency to manifold traversal and demonstrates this behavior through experiments on real datasets and attacks.

Findings

Query efficiency correlates with closer proximity to the data manifold.

Zeroth-order attacks can produce samples with reduced manifold distance.

Manifold-gradient mutual information can guide robust model development.

Abstract

Designing deep networks robust to adversarial examples remains an open problem. Likewise, recent zeroth order hard-label attacks on image classification models have shown comparable performance to their first-order, gradient-level alternatives. It was recently shown in the gradient-level setting that regular adversarial examples leave the data manifold, while their on-manifold counterparts are in fact generalization errors. In this paper, we argue that query efficiency in the zeroth-order setting is connected to an adversary's traversal through the data manifold. To explain this behavior, we propose an information-theoretic argument based on a noisy manifold distance oracle, which leaks manifold information through the adversary's gradient estimate. Through numerical experiments of manifold-gradient mutual information, we show this behavior acts as a function of the effective problem dimensionality and number of training points. On real-world datasets and multiple zeroth-order attacks using dimension-reduction, we observe the same universal behavior to produce samples closer to the data manifold. This results in up to two-fold decrease in the manifold distance measure, regardless of the model robustness. Our results suggest that taking the manifold-gradient mutual information into account can thus inform better robust model design in the future, and avoid leakage of the sensitive data manifold.