PROV-AGENT: Unified Provenance for Tracking AI Agent Interactions in Agentic Workflows

TL;DR

PROV-AGENT is a provenance framework that captures and relates AI agent interactions within workflows, enhancing transparency, traceability, and reliability in complex, heterogeneous environments.

Contribution

It introduces a novel provenance model tailored for AI agent workflows, along with an open-source system for real-time provenance capture and evaluation across diverse computing environments.

Findings

Supports critical provenance queries for AI workflows

Demonstrates agent reliability analysis in real-world settings

Works across edge, cloud, and HPC environments

Abstract

Large Language Models (LLMs) and other foundation models are increasingly used as the core of AI agents. In agentic workflows, these agents plan tasks, interact with humans and peers, and influence scientific outcomes across federated and heterogeneous environments. However, agents can hallucinate or reason incorrectly, propagating errors when one agent's output becomes another's input. Thus, assuring that agents' actions are transparent, traceable, reproducible, and reliable is critical to assess hallucination risks and mitigate their workflow impacts. While provenance techniques have long supported these principles, existing methods fail to capture and relate agent-centric metadata such as prompts, responses, and decisions with the broader workflow context and downstream outcomes. In this paper, we introduce PROV-AGENT, a provenance model that extends W3C PROV and leverages the Model Context Protocol (MCP) and data observability to integrate agent interactions into end-to-end workflow provenance. Our contributions include: (1) a provenance model tailored for agentic workflows, (2) a near real-time, open-source system for capturing agentic provenance, and (3) a cross-facility evaluation spanning edge, cloud, and HPC environments, demonstrating support for critical provenance queries and agent reliability analysis.