Semantically Guided Adversarial Testing of Vision Models Using Language Models

TL;DR

This paper introduces a semantics-guided framework for selecting adversarial targets in vision model attacks using pretrained language models, improving interpretability and effectiveness over traditional methods.

Contribution

It proposes a novel cross-modal knowledge transfer approach leveraging pretrained language and vision-language models for targeted adversarial testing.

Findings

Pretrained models outperform static lexical resources in target selection.

Semantic similarity sources influence attack success and target relevance.

Framework enables scalable, interpretable adversarial benchmarks.

Abstract

In targeted adversarial attacks on vision models, the selection of the target label is a critical yet often overlooked determinant of attack success. This target label corresponds to the class that the attacker aims to force the model to predict. Now, existing strategies typically rely on randomness, model predictions, or static semantic resources, limiting interpretability, reproducibility, or flexibility. This paper then proposes a semantics-guided framework for adversarial target selection using the cross-modal knowledge transfer from pretrained language and vision-language models. We evaluate several state-of-the-art models (BERT, TinyLLAMA, and CLIP) as similarity sources to select the most and least semantically related labels with respect to the ground truth, forming best- and worst-case adversarial scenarios. Our experiments on three vision models and five attack methods reveal that these models consistently render practical adversarial targets and surpass static lexical databases, such as WordNet, particularly for distant class relationships. We also observe that static testing of target labels offers a preliminary assessment of the effectiveness of similarity sources, \textit{a priori} testing. Our results corroborate the suitability of pretrained models for constructing interpretable, standardized, and scalable adversarial benchmarks across architectures and datasets.