Credential Leakage in LLM Agent Skills: A Large-Scale Empirical Study

TL;DR

This large-scale empirical study investigates credential leakage in third-party LLM agent skills, revealing common patterns, vulnerabilities, and mitigation outcomes, supported by a dataset and detection tools.

Contribution

First comprehensive analysis of credential leakage in LLM skills, identifying key patterns, vulnerabilities, and providing datasets and tools for future research.

Findings

76.3% leakage involves cross-modal analysis of code and natural language

73.5% of leaks caused by debug logging like print and console.log

89.6% of leaked credentials are exploitable without privileges

Abstract

Third-party skills extend LLM agents with powerful capabilities but often handle sensitive credentials in privileged environments, making leakage risks poorly understood. We present the first large-scale empirical study of this problem, analyzing 17,022 skills (sampled from 170,226 on SkillsMP) using static analysis, sandbox testing, and manual inspection. We identify 520 vulnerable skills with 1,708 issues and derive a taxonomy of 10 leakage patterns (4 accidental and 6 adversarial). We find that (1) leakage is fundamentally cross-modal: 76.3% require joint analysis of code and natural language, while 3.1% arise purely from prompt injection; (2) debug logging is the primary vector, with print and console.log causing 73.5% of leaks due to stdout exposure to LLMs; and (3) leaked credentials are both exploitable (89.6% without privileges) and persistent, as forks retain secrets even after upstream fixes. After disclosure, all malicious skills were removed and 91.6% of hardcoded credentials were fixed. We release our dataset, taxonomy, and detection pipeline to support future research.