# FHIR-AgentEval: A Modular Sandbox for Benchmarking Clinical LLM Agents with an Evaluation of Memory-Augmented Configurations

## TL;DR

This paper introduces FHIR-AgentEval, a tool for testing AI systems in healthcare workflows, showing that memory features improve performance.

## Contribution

The novel contribution is FHIR-AgentEval, a modular benchmark for evaluating clinical LLM agents with memory-augmented configurations.

## Key findings

- Memory-augmented agents improved task success by 9.1% over baselines.
- Memory reduced strategic failures like incorrect tool selection.
- The sandbox supports 43 modular tasks for realistic clinical workflows.

## Abstract

Healthcare data exchange increasingly relies on HL7 FHIR, but FHIR’s implementation complexity creates barriers for clinical workflows. Large language model (LLM) agents could bridge this gap by translating natural language requests into structured FHIR operations, yet their reliability remains unproven. We present FHIR-AgentEval, an extensible evaluation sandbox comprising 43 modular tasks for benchmarking LLM agents on realistic appointment management and genetic testing workflows. Each task executes against a resettable FHIR server with custom deterministic validation of both agent responses and resulting server state. We run an ablation study of five agent configurations, varying access to an on-demand FHIR R4 specifications server and long-term memory trained with or without specification grounding. Across four experimental settings, memory consistently improves task success and reduces strategic failures such as incorrect tool selection and resource-type confusion. On held-out tasks, the best memory configuration improves success by 9.1% over baseline, offering a potential pathway toward more robust clinical deployment.