# MedFoundationHub: A Lightweight and Secure Toolkit for Deploying Medical Vision Language Foundation Models

## TL;DR

MedFoundationHub is a secure, user-friendly toolkit that enables medical professionals and engineers to deploy vision-language models efficiently while safeguarding patient data in clinical environments.

## Contribution

The paper introduces MedFoundationHub, a GUI toolkit that simplifies deployment of medical VLMs with privacy safeguards, requiring minimal resources and no programming expertise.

## Key findings

- Evaluated five state-of-the-art VLMs with clinicians, resulting in 1015 scoring events.
- Identified limitations such as off-target answers and vague reasoning in current models.
- Demonstrated the toolkit's effectiveness in clinical model deployment and assessment.

## Abstract

Recent advances in medical vision-language models (VLMs) open up remarkable opportunities for clinical applications such as automated report generation, copilots for physicians, and uncertainty quantification. However, despite their promise, medical VLMs introduce serious security concerns, most notably risks of Protected Health Information (PHI) exposure, data leakage, and vulnerability to cyberthreats - which are especially critical in hospital environments. Even when adopted for research or non-clinical purposes, healthcare organizations must exercise caution and implement safeguards. To address these challenges, we present MedFoundationHub, a graphical user interface (GUI) toolkit that: (1) enables physicians to manually select and use different models without programming expertise, (2) supports engineers in efficiently deploying medical VLMs in a plug-and-play fashion, with seamless integration of Hugging Face open-source models, and (3) ensures privacy-preserving inference through Docker-orchestrated, operating system agnostic deployment. MedFoundationHub requires only an offline local workstation equipped with a single NVIDIA A6000 GPU, making it both secure and accessible within the typical resources of academic research labs. To evaluate current capabilities, we engaged board-certified pathologists to deploy and assess five state-of-the-art VLMs (Google-MedGemma3-4B, Qwen2-VL-7B-Instruct, Qwen2.5-VL-7B-Instruct, and LLaVA-1.5-7B/13B). Expert evaluation covered colon cases and renal cases, yielding 1015 clinician-model scoring events. These assessments revealed recurring limitations, including off-target answers, vague reasoning, and inconsistent pathology terminology.