Gradient Structure Estimation under Label-Only Oracles via Spectral Sensitivity

TL;DR

This paper introduces a new gradient estimation attack in hard-label black-box settings, combining spectral initialization and pattern-driven optimization, achieving state-of-the-art success and query efficiency across various models and tasks.

Contribution

It provides a theoretical framework linking existing sign-flipping attacks to gradient sign recovery and proposes a novel attack method with proven guarantees and superior empirical performance.

Findings

Outperforms state-of-the-art hard-label attacks in success rate and query efficiency.

Effective across multiple datasets and model types, including defenses.

Successfully bypasses advanced defenses like Blacklight.

Abstract

Hard-label black-box settings, where only top-1 predicted labels are observable, pose a fundamentally constrained yet practically important feedback model for understanding model behavior. A central challenge in this regime is whether meaningful gradient information can be recovered from such discrete responses. In this work, we develop a unified theoretical perspective showing that a wide range of existing sign-flipping hard-label attacks can be interpreted as implicitly approximating the sign of the true loss gradient. This observation reframes hard-label attacks from heuristic search procedures into instances of gradient sign recovery under extremely limited feedback. Motivated by this first-principles understanding, we propose a new attack framework that combines a zero-query frequency-domain initialization with a Pattern-Driven Optimization (PDO) strategy. We establish theoretical guarantees demonstrating that, under mild assumptions, our initialization achieves higher expected cosine similarity to the true gradient sign compared to random baselines, while the proposed PDO procedure attains substantially lower query complexity than existing structured search approaches. We empirically validate our framework through extensive experiments on CIFAR-10, ImageNet, and ObjectNet, covering standard and adversarially trained models, commercial APIs, and CLIP-based models. The results show that our method consistently surpasses SOTA hard-label attacks in both attack success rate and query efficiency, particularly in low-query regimes. Beyond image classification, our approach generalizes effectively to corrupted data, biomedical datasets, and dense prediction tasks. Notably, it also successfully circumvents Blacklight, a SOTA stateful defense, resulting in a $0\%$ detection rate. Our code will be released publicly soon at https://github.com/csjunjun/DPAttack.git.