ProtoMedAgent: Multimodal Clinical Interpretability via Privacy-Aware Agentic Workflows

TL;DR

ProtoMedAgent is a novel multimodal clinical reporting framework that enhances interpretability and privacy by combining neuro-symbolic reasoning, set-theoretic constraints, and privacy safeguards, outperforming standard retrieval methods.

Contribution

It introduces a zero-gradient, test-time optimization approach over a neuro-symbolic backbone for clinical interpretability with privacy guarantees.

Findings

Achieves 91.2% faithfulness in clinical reports, outperforming standard RAG.

Reduces artifact-level membership inference risk by 9.8%.

Effectively integrates visual and tabular data into semantic clinical narratives.

Abstract

While interpretable prototype networks offer compelling case-based reasoning for clinical diagnostics, their raw continuous outputs lack the semantic structure required for medical documentation. Bridging this gap via standard Retrieval-Augmented Generation (RAG) routinely triggers ``retrieval sycophancy,'' where Large Language Models (LLMs) hallucinate post-hoc rationalizations to align with visual predictions. We introduce ProtoMedAgent, a framework that formalizes multimodal clinical reporting as an iterative, zero-gradient test-time optimization problem over a strict neuro-symbolic bottleneck. Operating on a frozen prototype backbone, we distill latent visual and tabular features into a discrete semantic memory. Online generation is strictly constrained by exact set-theoretic differentials and a reflective Scribe-Critic loop, mathematically precluding unsupported narrative claims. To safely bound data disclosure, we introduce a semantic privacy gate governed by $k$-anonymity and $\ell$-diversity. Evaluated on a 4,160-patient clinical cohort, ProtoMedAgent achieves 91.2\% Comparison Set Faithfulness where it fundamentally outperforms standard RAG (46.2\%). ProtoMedAgent additionally leverages a binding $\ell$-diversity phase transition to systematically reduce artifact-level membership inference risks by an absolute 9.8\%.