ARMOR: Agentic Reasoning for Methods Orchestration and Reparameterization for Robust Adversarial Attacks

TL;DR

ARMOR introduces an adaptive, agent-based framework that orchestrates multiple adversarial attack methods using vision and language models, significantly improving attack transferability and success rates on standard benchmarks.

Contribution

This work presents ARMOR, a novel framework that dynamically orchestrates and reparameterizes multiple attack methods using LLM-guided agents for enhanced adversarial attack effectiveness.

Findings

Improved cross-architecture transferability of attacks.

Reliable fooling of models in both black-box and white-box settings.

Effective blending and selection of attack strategies based on confidence and SSIM scores.

Abstract

Existing automated attack suites operate as static ensembles with fixed sequences, lacking strategic adaptation and semantic awareness. This paper introduces the Agentic Reasoning for Methods Orchestration and Reparameterization (ARMOR) framework to address these limitations. ARMOR orchestrates three canonical adversarial primitives, Carlini-Wagner (CW), Jacobian-based Saliency Map Attack (JSMA), and Spatially Transformed Attacks (STA) via Vision Language Models (VLM)-guided agents that collaboratively generate and synthesize perturbations through a shared ``Mixing Desk". Large Language Models (LLMs) adaptively tune and reparameterize parallel attack agents in a real-time, closed-loop system that exploits image-specific semantic vulnerabilities. On standard benchmarks, ARMOR achieves improved cross-architecture transfer and reliably fools both settings, delivering a blended output for blind targets and selecting the best attack or blended attacks for white-box targets using a confidence-and-SSIM score.