An Early Investigation into the Utility of Multimodal Large Language Models in Medical Imaging

TL;DR

This study evaluates the capabilities of multimodal large language models Gemini and GPT-4V in classifying and interpreting real versus synthetic medical images, revealing their potential and limitations in medical imaging analysis.

Contribution

It provides an early assessment of MLLMs' utility in medical image classification and interpretation, highlighting their comparative performance and current limitations.

Findings

Gemini slightly outperformed GPT-4V in classification tasks.

GPT-4V responses were mostly generic and less specific.

Both models showed potential in interpreting medical images.

Abstract

Recent developments in multimodal large language models (MLLMs) have spurred significant interest in their potential applications across various medical imaging domains. On the one hand, there is a temptation to use these generative models to synthesize realistic-looking medical image data, while on the other hand, the ability to identify synthetic image data in a pool of data is also significantly important. In this study, we explore the potential of the Gemini (\textit{gemini-1.0-pro-vision-latest}) and GPT-4V (gpt-4-vision-preview) models for medical image analysis using two modalities of medical image data. Utilizing synthetic and real imaging data, both Gemini AI and GPT-4V are first used to classify real versus synthetic images, followed by an interpretation and analysis of the input images. Experimental results demonstrate that both Gemini and GPT-4 could perform some interpretation of the input images. In this specific experiment, Gemini was able to perform slightly better than the GPT-4V on the classification task. In contrast, responses associated with GPT-4V were mostly generic in nature. Our early investigation presented in this work provides insights into the potential of MLLMs to assist with the classification and interpretation of retinal fundoscopy and lung X-ray images. We also identify key limitations associated with the early investigation study on MLLMs for specialized tasks in medical image analysis.