SkillProbe: Security Auditing for Emerging Agent Skill Marketplaces via Multi-Agent Collaboration

TL;DR

SkillProbe is a multi-agent security auditing framework for emerging LLM agent skill marketplaces, addressing behavioral and combinatorial risks through standardized, collaborative auditing processes, revealing systemic security issues in popular skills.

Contribution

We introduce SkillProbe, a novel multi-stage, multi-agent security auditing system utilizing a standardized 'Skills-for-Skills' paradigm to detect behavioral and combinatorial risks in skill marketplaces.

Findings

Over 90% of high-popularity skills failed security audits.

High-risk skills form a giant connected component, indicating systemic risks.

Download volume is unreliable as a proxy for skill security quality.

Abstract

With the rapid evolution of Large Language Model (LLM) agent ecosystems, centralized skill marketplaces have emerged as pivotal infrastructure for augmenting agent capabilities. However, these marketplaces face unprecedented security challenges, primarily stemming from semantic-behavioral inconsistency and inter-skill combinatorial risks, where individually benign skills induce malicious behaviors during collaborative invocation. To address these vulnerabilities, we propose SkillProbe, a multi-stage security auditing framework driven by multi-agent collaboration. SkillProbe introduces a "Skills-for-Skills" design paradigm, encapsulating auditing processes into standardized skill modules to drive specialized agents through a rigorous pipeline, including admission filtering, semantic-behavioral alignment detection, and combinatorial risk simulation. We conducted a large-scale evaluation using 8 mainstream LLM series across 2,500 real-world skills from ClawHub. Our results reveal a striking popularity-security paradox, where download volume is not a reliable proxy for security quality, as over 90% of high-popularity skills failed to pass rigorous auditing. Crucially, we discovered that high-risk skills form a single giant connected component within the risk-link dimension, demonstrating that cascaded risks are systemic rather than isolated occurrences. We hope that SkillProbe will inspire researchers to provide a scalable governance infrastructure for constructing a trustworthy Agentic Web. SkillProbe is accessible for public experience at skillhub.holosai.io.