MiniGPT-Pancreas: Multimodal Large Language Model for Pancreas Cancer Observation and Localization in CT Images

TL;DR

MiniGPT-Pancreas is a lightweight AI tool that helps detect and locate pancreas and tumors in CT scans using natural language and outperforms other models in accuracy and usability.

Contribution

Introduces MiniGPT-Pancreas, a compact MLLM for pancreas and tumor detection in CT images that outperforms larger models in performance and usability.

Findings

MiniGPT-Pancreas achieved an average IoU of 0.57 for pancreas detection, surpassing other models like GLM-4.1V-9B-Base and UMIT.

Tumor observation on the MSD dataset yielded an accuracy, precision, recall, and F1 score of about 0.87, outperforming existing models.

Expert surgeons rated MiniGPT-Pancreas highly for training and verification, with scores of 4.5/5.0 on a 5-point scale.

Abstract

Pancreatic cancer remains one of the deadliest malignancies, primarily because of its subtle CT appearance and frequent late-stage diagnosis. We introduce MiniGPT-Pancreas, a lightweight multimodal large language model (MLLM) that interprets natural-language queries within an interactive ChatGPT-style interface, as well as computed tomography images, and returns precise bounding-box predictions for the pancreas and associated tumors. A cascaded fine-tuning strategy was applied to MiniGPTv2, a multi-task general-purpose MLLM, with a focus on pancreas and tumor detection, using the National Institute of Health (NIH) and Medical Segmentation Decathlon (MSD) pancreas datasets. Pancreas detection achieved an average intersection over Union (IoU) of 0.57 on NIH and MSD datasets, outperforming the base MiniGPT-Pancreas model and more recent MLLMs like GLM-4.1V-9B-Base (general-purpose) and UMIT (specific to the biomedical domain). Tumor observation on MSD yielded an accuracy, precision, recall, and F1 score of all about 0.87, surpassing MiniGPT-v2, GLM-4.1V-9B-Base, and UMIT. For tumor localization, the IoU was 0.28, higher than UMIT (IoU=0.07), but lower than GLM-4.1V-9B-Base (IoU=0.48). On multi-organ detection on the AbdomenCT-1k dataset, MiniGPT-Pancreas outperformed GLM-4.1V-9B-Base and UMIT in all organs, with an IoU of 0.50 on pancreas vs. 0.43 and 0.03, respectively. MiniGPT-Pancreas was rated highly by an international group of 10 expert general surgeons (Italy, Singapore, and the UK) as a potential training tool, especially for verification (4.5/5.0), and training of young specialists (4.5/5.0) on a 5-point Likert scale. While operating on 2D slices limits volumetric context, MiniGPT-Pancreas demonstrates that compact MLLMs can rival specialized vision networks in pancreas imaging, offering an intuitive, language-driven tool for AI-assisted radiology. The code is publicly available at https://github.com/elianastasio/MiniGPTPancreas.

Figures