Sparse Spectral LoRA: Routed Experts for Medical VLMs

TL;DR

This paper introduces MedQwen, a parameter-efficient medical vision-language model using spectrally routed experts, achieving high performance with minimal training and reduced forgetting across diverse medical tasks.

Contribution

It proposes a novel spectrally routed Mixture-of-Experts approach with a scaling rule, enabling efficient and stable continual learning in medical VLMs.

Findings

Approaches full fine-tuning performance with 339× fewer parameters.

Reduces sequential forgetting to approximately 5%.

Achieves strong results across 23 medical datasets.

Abstract

Large vision-language models (VLMs) excel on general benchmarks but often lack robustness in medical imaging, where heterogeneous supervision induces cross-dataset interference and sensitivity to data regime (i.e., how the supervisory signals are mixed). In realistic clinical workflows, data and tasks arrive sequentially, so naive continual training further leads to catastrophic forgetting. To address these challenges, we propose MedQwen, a parameter-efficient medical VLM that couples a spectrally routed Mixture-of-Experts (MoE) with a theoretically grounded scaling rule that aligns low-rank updates with a full-rank, fully fine-tuned MoE, without changing the base architecture. Concretely, we initialize each expert from non-overlapping singular value decomposition (SVD) segments of the pretrained weight and introduce a residual compensation and scaling scheme to enable stable expert specialization and consistent routing under distribution shift. Across 23 medical datasets covering visual question answering, report generation, radiology classification, and hallucination mitigation, MedQwen achieves strong, reliable performance: it approaches full fine-tuning on zero-shot classification with 339$\times$ fewer trainable parameters, and reduces sequential forgetting to $\sim$5\% where strong baselines degrade by $>$20-50\%.