MiniGPT-Pancreas: Multimodal Large Language Model for Pancreas Cancer Classification and Detection

TL;DR

MiniGPT-Pancreas is a multimodal large language model designed to assist clinicians in pancreas cancer diagnosis by integrating visual and textual data, showing promising results in detection and classification tasks.

Contribution

This work introduces MiniGPT-Pancreas, a fine-tuned multimodal model specifically adapted for pancreas cancer detection and classification using multimodal prompts and medical imaging datasets.

Findings

Achieved IoU of 0.595 for pancreas detection on NIH dataset.

Achieved 87.6% accuracy in pancreas cancer classification.

IoU of 0.168 for pancreas tumor detection on MSD dataset.

Abstract

Problem: Pancreas radiological imaging is challenging due to the small size, blurred boundaries, and variability of shape and position of the organ among patients. Goal: In this work we present MiniGPT-Pancreas, a Multimodal Large Language Model (MLLM), as an interactive chatbot to support clinicians in pancreas cancer diagnosis by integrating visual and textual information. Methods: MiniGPT-v2, a general-purpose MLLM, was fine-tuned in a cascaded way for pancreas detection, tumor classification, and tumor detection with multimodal prompts combining questions and computed tomography scans from the National Institute of Health (NIH), and Medical Segmentation Decathlon (MSD) datasets. The AbdomenCT-1k dataset was used to detect the liver, spleen, kidney, and pancreas. Results: MiniGPT-Pancreas achieved an Intersection over Union (IoU) of 0.595 and 0.550 for the detection of pancreas on NIH and MSD datasets, respectively. For the pancreas cancer classification task on the MSD dataset, accuracy, precision, and recall were 0.876, 0.874, and 0.878, respectively. When evaluating MiniGPT-Pancreas on the AbdomenCT-1k dataset for multi-organ detection, the IoU was 0.8399 for the liver, 0.722 for the kidney, 0.705 for the spleen, and 0.497 for the pancreas. For the pancreas tumor detection task, the IoU score was 0.168 on the MSD dataset. Conclusions: MiniGPT-Pancreas represents a promising solution to support clinicians in the classification of pancreas images with pancreas tumors. Future research is needed to improve the score on the detection task, especially for pancreas tumors.