AURA: A Multi-Modal Medical Agent for Understanding, Reasoning & Annotation

TL;DR

AURA is a novel multi-modal AI agent designed for medical image analysis that combines visual and linguistic explanations, enabling interactive, explainable, and clinically relevant reasoning in medical diagnostics.

Contribution

This work introduces AURA, the first visual linguistic explainability agent tailored for medical imaging, integrating multiple modules for segmentation, reasoning, and evaluation.

Findings

AURA enables dynamic interaction and hypothesis testing in medical image analysis.

AURA provides transparent, interpretable explanations for medical diagnoses.

AURA demonstrates improved diagnostic relevance and visual interpretability.

Abstract

Recent advancements in Large Language Models (LLMs) have catalyzed a paradigm shift from static prediction systems to agentic AI agents capable of reasoning, interacting with tools, and adapting to complex tasks. While LLM-based agentic systems have shown promise across many domains, their application to medical imaging remains in its infancy. In this work, we introduce AURA, the first visual linguistic explainability agent designed specifically for comprehensive analysis, explanation, and evaluation of medical images. By enabling dynamic interactions, contextual explanations, and hypothesis testing, AURA represents a significant advancement toward more transparent, adaptable, and clinically aligned AI systems. We highlight the promise of agentic AI in transforming medical image analysis from static predictions to interactive decision support. Leveraging Qwen-32B, an LLM-based architecture, AURA integrates a modular toolbox comprising: (i) a segmentation suite with phase grounding, pathology segmentation, and anatomy segmentation to localize clinically meaningful regions; (ii) a counterfactual image-generation module that supports reasoning through image-level explanations; and (iii) a set of evaluation tools including pixel-wise difference-map analysis, classification, and advanced state-of-the-art components to assess diagnostic relevance and visual interpretability.