Toward User Comprehension Supports for LLM Agent Skill Specifications

TL;DR

This paper investigates how user-facing skill specifications for LLM agents can better support user comprehension by analyzing textual cues that indicate operational details, boundaries, and examples within skill descriptions.

Contribution

It highlights the scarcity of comprehensive cues in skill specifications and advocates for viewing these specifications as capability disclosures to improve user understanding.

Findings

Operational basis cues are common in specifications.

Only 19% of specifications include example tasks or outcomes.

Missing examples hinder users' ability to verify and understand skills.

Abstract

Users often interpret and select agent skills through their SKILL markdown specifications. To protect users, existing audits mainly focus on malicious or unsafe skills. We study the complementary question of whether specifications help users form bounded expectations about what a skill consumes, produces, and covers. Across 878 cybersecurity skills, we used rule-based coding to measure textual cues for four comprehension anchors, namely operational basis, output contract, boundary disclosure, and example capability demonstration. Cues for operational basis were common, but only 19.0% of specifications exhibited cues for an example task, sample, or expected outcome, and only 2.3% exhibited cues for all four anchors. We further examined a small DNS/C2 telemetry subset (n$=$6) to illustrate why missing examples may matter. Examples appeared to make first local checks easier to construct, while no-example skills typically required helper code inspection to recover command arguments or output fields. We argue that agent-skill evaluation should treat specifications as user-facing capability disclosures, not merely as containers for executable instructions.