MedPrompt: LLM-CNN Fusion with Weight Routing for Medical Image Segmentation and Classification

TL;DR

MedPrompt is a unified framework that combines large language models and modular CNNs to enable flexible, prompt-driven medical image analysis across multiple tasks and modalities, with high accuracy and efficiency.

Contribution

The paper introduces MedPrompt, a novel system that integrates LLM-based task planning with weight routing in CNNs, allowing scalable multi-task medical imaging without retraining.

Findings

Achieves 97% correctness in task interpretation and execution.

Demonstrates high segmentation accuracy (Dice 0.9856) and classification F1 (0.9744).

Operates with an average latency of 2.5 seconds, suitable for near real-time use.

Abstract

Current medical image analysis systems are typically task-specific, requiring separate models for classification and segmentation, and lack the flexibility to support user-defined workflows. To address these challenges, we introduce MedPrompt, a unified framework that combines a few-shot prompted Large Language Model (Llama-4-17B) for high-level task planning with a modular Convolutional Neural Network (DeepFusionLab) for low-level image processing. The LLM interprets user instructions and generates structured output to dynamically route task-specific pretrained weights. This weight routing approach avoids retraining the entire framework when adding new tasks-only task-specific weights are required, enhancing scalability and deployment. We evaluated MedPrompt across 19 public datasets, covering 12 tasks spanning 5 imaging modalities. The system achieves a 97% end-to-end correctness in interpreting and executing prompt-driven instructions, with an average inference latency of 2.5 seconds, making it suitable for near real-time applications. DeepFusionLab achieves competitive segmentation accuracy (e.g., Dice 0.9856 on lungs) and strong classification performance (F1 0.9744 on tuberculosis). Overall, MedPrompt enables scalable, prompt-driven medical imaging by combining the interpretability of LLMs with the efficiency of modular CNNs.