Semia: Auditing Agent Skills via Constraint-Guided Representation Synthesis

TL;DR

Semia is a static auditing tool that uses constraint-guided synthesis to verify security properties of agent skills expressed in a hybrid language, identifying critical risks in real-world skills.

Contribution

It introduces CGRS, a novel loop for synthesizing faithful representations of skills, enabling effective security analysis of complex agent capabilities.

Findings

Semia successfully audits 13,728 real-world skills from public marketplaces.

More than half of the skills contain at least one critical semantic security risk.

Semia outperforms existing signature-based and LLM baselines with 97.7% recall and 90.6% F1.

Abstract

An agent skill is a configuration package that equips an LLM-driven agent with a concrete capability, such as reading email, executing shell commands, or signing blockchain transactions. Each skill is a hybrid artifact-a structured half declares executable interfaces, while a prose half dictates when and how those interfaces fire-and the prose is reinterpreted probabilistically on every invocation. Conventional static analyzers parse the structured half but ignore the prose; LLM-based tools read the prose but cannot reproducibly prove that a tainted input reaches a high-impact sink. We present Semia, a static auditor for agent skills. Semia lifts each skill into the Skill Description Language (SDL), a Datalog fact base that captures LLM-triggered actions, prose-defined conditions, and human-in-the-loop checkpoints. Synthesizing a fact base that is both structurally sound and semantically faithful to the original prose is the central challenge; we address it with Constraint-Guided Representation Synthesis (CGRS), a propose-verify-evaluate loop that refines LLM candidates until convergence. Security properties (e.g., indirect injection, secret leakage, confused deputies, unguarded sinks, etc.) over an agent skill can then be reduced to Datalog reachability queries. We evaluate Semia on 13,728 real-world skills from public marketplaces. Semia renders all of them auditable and finds that more than half carry at least one critical semantic risk. On a stratified sample of 541 expert-labeled skills, Semia achieves 97.7% recall and an F1 of 90.6%, substantially outperforming signature-based scanners and LLM baselines.