MedOpenClaw and MedFlowBench: Auditing Medical Agents in Full-Study Workflows

TL;DR

This paper introduces MedFlowBench and MedOpenClaw to evaluate medical imaging agents on full-study workflows, emphasizing the importance of producing auditable evidence in clinical settings.

Contribution

It presents a new benchmark and runtime environment for assessing whether medical imaging agents can operate on complete studies and produce verifiable evidence.

Findings

Answer-only scoring overestimates model performance.

Adding image-analysis tools alone does not solve workflow challenges.

Models struggle with input selection, viewer management, and evidence verification.

Abstract

Medical imaging benchmarks often evaluate VLMs on pre-selected 2D images, slices, crops, or patches, making evaluation closer to visual recognition. Real clinical workflows impose a different burden: readers must search through complete studies, operate imaging software, navigate across slices and magnifications, and document visual evidence that can be audited. We argue that this evidence-producing workflow is a critical missing evaluation axis for medical imaging agents. To study it, we introduce MedFlowBench, a full-study benchmark for VLM agents, together with MedOpenClaw, a controlled and replayable runtime in which agents operate medical imaging viewers such as 3D Slicer and QuPath. In each episode, an agent inspects a complete radiology study or whole-slide pathology image, returns a task answer, and submits structured evidence, including key slices, coordinates, regions of interest, or lesion-state fields. This evidence is automatically checked against withheld masks, annotations, and labels. Across evaluated models, final answer-only scoring gives an overly optimistic picture: when answers must also be supported by correct evidence, performance drops substantially on complex workflows. We further find that adding image-analysis tools does not by itself solve the problem. Tools help when they make a complex procedure simple and reliable, but agents still struggle when they must choose inputs, manage viewer state, and verify intermediate outputs over multiple steps. MedFlowBench exposes whether medical imaging agents can produce auditable evidence from complete studies, rather than plausible answers from selected images.