SkillAttack: Automated Red Teaming of Agent Skills through Attack Path Refinement

TL;DR

SkillAttack is a framework that dynamically tests the security of open-source agent skills against adversarial prompts, revealing vulnerabilities without modifying the skills.

Contribution

It introduces a novel adversarial prompting approach for red-teaming agent skills, outperforming existing methods in vulnerability detection.

Findings

Achieved high attack success rates (0.73--0.93) on adversarial skills.

Detected vulnerabilities in 100 real-world skills.

Outperformed all baseline methods in experiments.

Abstract

LLM-based agent systems increasingly rely on agent skills sourced from open registries to extend their capabilities, yet the openness of such ecosystems makes skills difficult to thoroughly vet. Existing attacks rely on injecting malicious instructions into skills, making them easily detectable by static auditing. However, non-malicious skills may also harbor latent vulnerabilities that an attacker can exploit solely through adversarial prompting, without modifying the skill itself. We introduce SkillAttack, a red-teaming framework that dynamically verifies skill vulnerability exploitability through adversarial prompting. SkillAttack combines vulnerability analysis, surface-parallel attack generation, and feedback-driven exploit refinement into a closed-loop search that progressively converges toward successful exploitation. Experiments across 10 LLMs on 71 adversarial and 100 real-world skills show that SkillAttack outperforms all baselines by a wide margin (ASR 0.73--0.93 on adversarial skills, up to 0.26 on real-world skills), revealing that even well-intended skills pose serious security risks under realistic agent interactions.