Towards Verifiably Safe Tool Use for LLM Agents

TL;DR

This paper proposes a formal safety framework for LLM-based agents using system analysis and structured protocols to prevent unintended tool interactions and ensure enterprise-level security guarantees.

Contribution

It introduces a systematic hazard analysis and a capability-enhanced protocol to formalize safety requirements for LLM agents, moving beyond ad hoc safeguards.

Findings

Application of STPA to identify hazards in agent workflows

Development of a capability-enhanced Model Context Protocol (MCP) framework

Formalization of safety requirements as enforceable specifications

Abstract

Large language model (LLM)-based AI agents extend LLM capabilities by enabling access to tools such as data sources, APIs, search engines, code sandboxes, and even other agents. While this empowers agents to perform complex tasks, LLMs may invoke unintended tool interactions and introduce risks, such as leaking sensitive data or overwriting critical records, which are unacceptable in enterprise contexts. Current approaches to mitigate these risks, such as model-based safeguards, enhance agents' reliability but cannot guarantee system safety. Methods like information flow control (IFC) and temporal constraints aim to provide guarantees but often require extensive human annotation. We propose a process that starts with applying System-Theoretic Process Analysis (STPA) to identify hazards in agent workflows, derive safety requirements, and formalize them as enforceable specifications on data flows and tool sequences. To enable this, we introduce a capability-enhanced Model Context Protocol (MCP) framework that requires structured labels on capabilities, confidentiality, and trust level. Together, these contributions aim to shift LLM-based agent safety from ad hoc reliability fixes to proactive guardrails with formal guarantees, while reducing dependence on user confirmation and making autonomy a deliberate design choice.