Malicious Or Not: Adding Repository Context to Agent Skill Classification

TL;DR

This paper conducts the largest empirical security analysis of AI agent skills, reducing false positives in malicious classification by incorporating repository context and uncovering new attack vectors.

Contribution

It introduces a methodology that combines skill description analysis with repository context to improve malicious skill detection accuracy.

Findings

Reduces false positives from 46.8% to 0.52% in security scans.

Uncovers undocumented attack vectors like hijacked abandoned repositories.

Provides a comprehensive view of the current risk surface in AI agent ecosystems.

Abstract

Agent skills extend local AI agents, such as Claude Code or Open Claw, with additional functionality, and their popularity has led to the emergence of dedicated skill marketplaces, similar to app stores for mobile applications. Simultaneously, automated skill scanners were introduced, analyzing the skill description available in SKILL.md, to verify their benign behavior. The results for individual market places mark up to 46.8% of skills as malicious. In this paper, we present the largest empirical security analysis of the AI agent skill ecosystem, questioning this high classification of malicious skills. Therefore, we collect 238,180 unique skills from three major distribution platforms and GitHub to systematically analyze their type and behavior. This approach substantially reduces the number of skills flagged as non-benign by security scanners to only 0.52% which remain in malicious flagged repositories. Consequently, out methodology substantially reduces false positives and provides a more robust view of the ecosystem's current risk surface. Beyond that, we extend the security analysis from the mere investigation of the skill description to a comparison of its congruence with the GitHub repository the skill is embedded in, providing additional context. Furthermore, our analysis also uncovers several, by now undocumented real-world attack vectors, namely hijacking skills hosted on abandoned GitHub repositories.