VSF-Med:A Vulnerability Scoring Framework for Medical Vision-Language Models

TL;DR

This paper presents VSF-Med, a comprehensive vulnerability scoring framework for medical vision-language models, combining attack templates, imperceptible perturbations, and a multi-dimensional rubric to evaluate security risks systematically.

Contribution

The paper introduces VSF-Med, the first end-to-end framework for systematically assessing vulnerabilities in medical VLMs using novel attack methods and a standardized scoring rubric.

Findings

State-of-the-art VLMs show significant vulnerability increases under attack.

VSF-Med enables reproducible benchmarking with over 30,000 adversarial variants.

Llama-3.2-11B-Vision-Instruct exhibits the highest vulnerability among tested models.

Abstract

Vision Language Models (VLMs) hold great promise for streamlining labour-intensive medical imaging workflows, yet systematic security evaluations in clinical settings remain scarce. We introduce VSF--Med, an end-to-end vulnerability-scoring framework for medical VLMs that unites three novel components: (i) a rich library of sophisticated text-prompt attack templates targeting emerging threat vectors; (ii) imperceptible visual perturbations calibrated by structural similarity (SSIM) thresholds to preserve clinical realism; and (iii) an eight-dimensional rubric evaluated by two independent judge LLMs, whose raw scores are consolidated via z-score normalization to yield a 0--32 composite risk metric. Built entirely on publicly available datasets and accompanied by open-source code, VSF--Med synthesizes over 30,000 adversarial variants from 5,000 radiology images and enables reproducible benchmarking of any medical VLM with a single command. Our consolidated analysis reports mean z-score shifts of $0.90\sigma$ for persistence-of-attack-effects, $0.74\sigma$ for prompt-injection effectiveness, and $0.63\sigma$ for safety-bypass success across state-of-the-art VLMs. Notably, Llama-3.2-11B-Vision-Instruct exhibits a peak vulnerability increase of $1.29\sigma$ for persistence-of-attack-effects, while GPT-4o shows increases of $0.69\sigma$ for that same vector and $0.28\sigma$ for prompt-injection attacks.