VIVID-Med: LLM-Supervised Structured Pretraining for Deployable Medical ViTs

TL;DR

VIVID-Med introduces a novel LLM-supervised pretraining framework for medical vision transformers, effectively capturing complex clinical semantics and enabling deployable, resource-efficient models with superior performance across multiple medical imaging tasks.

Contribution

It presents a structured semantic supervision method using a frozen LLM and a new training scheme, significantly improving medical ViT pretraining and deployment efficiency.

Findings

Achieves 0.8588 macro-AUC on CheXpert, outperforming BiomedCLIP by +6.65 points.

Demonstrates strong zero-shot transfer to NIH ChestX-ray14 with 0.7225 macro-AUC.

Excels in cross-modality tasks, with 0.8413 AUC on LIDC-IDRI and 0.9969 on OrganAMNIST.

Abstract

Vision-language pretraining has driven significant progress in medical image analysis. However, current methods typically supervise visual encoders using one-hot labels or free-form text, neither of which effectively captures the complex semantic relationships among clinical findings. In this study, we introduce VIVID-Med, a novel framework that leverages a frozen large language model (LLM) as a structured semantic teacher to pretrain medical vision transformers (ViTs). VIVID-Med translates clinical findings into verifiable JSON field-state pairs via a Unified Medical Schema (UMS), utilizing answerability-aware masking to focus optimization. It then employs Structured Prediction Decomposition (SPD) to partition cross-attention into orthogonality-regularized query groups, extracting complementary visual aspects. Crucially, the LLM is discarded post-training, yielding a lightweight, deployable ViT-only backbone. We evaluated VIVID-Med across multiple settings: on CheXpert linear probing, it achieves a macro-AUC of 0.8588, outperforming BiomedCLIP by +6.65 points while using 500x less data. It also demonstrates robust zero-shot cross-domain transfer to NIH ChestX-ray14 (0.7225 macro-AUC) and strong cross-modality generalization to CT, achieving 0.8413 AUC on LIDC-IDRI lung nodule classification and 0.9969 macro-AUC on OrganAMNIST 11-organ classification. VIVID-Med offers a highly efficient, scalable alternative to deploying resource-heavy vision-language models in clinical settings.