Med-MoE: Mixture of Domain-Specific Experts for Lightweight Medical Vision-Language Models

TL;DR

Med-MoE introduces a lightweight, domain-specific mixture-of-experts framework for multimodal medical tasks, achieving high performance with fewer activated parameters, suitable for resource-constrained clinical settings.

Contribution

The paper presents a novel Med-MoE framework that efficiently handles both discriminative and generative medical tasks with fewer parameters, enhancing practicality in clinical applications.

Findings

Achieves superior or comparable performance to state-of-the-art models.

Requires only 30-50% of model parameters to be activated.

Demonstrates effectiveness across multiple medical datasets and tasks.

Abstract

Recent advancements in general-purpose or domain-specific multimodal large language models (LLMs) have witnessed remarkable progress for medical decision-making. However, they are designated for specific classification or generative tasks, and require model training or finetuning on large-scale datasets with sizeable parameters and tremendous computing, hindering their clinical utility across diverse resource-constrained scenarios in practice. In this paper, we propose a novel and lightweight framework Med-MoE (Mixture-of-Experts) that tackles both discriminative and generative multimodal medical tasks. The learning of Med-MoE consists of three steps: multimodal medical alignment, instruction tuning and routing, and domain-specific MoE tuning. After aligning multimodal medical images with LLM tokens, we then enable the model for different multimodal medical tasks with instruction tuning, together with a trainable router tailored for expert selection across input modalities. Finally, the model is tuned by integrating the router with multiple domain-specific experts, which are selectively activated and further empowered by meta expert. Comprehensive experiments on both open- and close-end medical question answering (Med-VQA) and image classification tasks across datasets such as VQA-RAD, SLAKE and Path-VQA demonstrate that our model can achieve performance superior to or on par with state-of-the-art baselines, while only requiring approximately 30\%-50\% of activated model parameters. Extensive analysis and ablations corroborate the effectiveness and practical utility of our method.