ChatCAD: Interactive Computer-Aided Diagnosis on Medical Image using Large Language Models

TL;DR

This paper introduces ChatCAD, a framework that integrates large language models with medical-image CAD networks to improve clinical decision support and patient communication by combining medical knowledge with image analysis.

Contribution

It presents a novel method for enhancing medical-image CAD systems with LLMs, enabling better interpretation and explanation of diagnostic information.

Findings

LLMs can effectively summarize and reorganize CAD network outputs.

The integrated system improves user understanding of medical diagnoses.

Potential for future improvements in vision-based medical-image analysis.

Abstract

Large language models (LLMs) have recently demonstrated their potential in clinical applications, providing valuable medical knowledge and advice. For example, a large dialog LLM like ChatGPT has successfully passed part of the US medical licensing exam. However, LLMs currently have difficulty processing images, making it challenging to interpret information from medical images, which are rich in information that supports clinical decisions. On the other hand, computer-aided diagnosis (CAD) networks for medical images have seen significant success in the medical field by using advanced deep-learning algorithms to support clinical decision-making. This paper presents a method for integrating LLMs into medical-image CAD networks. The proposed framework uses LLMs to enhance the output of multiple CAD networks, such as diagnosis networks, lesion segmentation networks, and report generation networks, by summarizing and reorganizing the information presented in natural language text format. The goal is to merge the strengths of LLMs' medical domain knowledge and logical reasoning with the vision understanding capability of existing medical-image CAD models to create a more user-friendly and understandable system for patients compared to conventional CAD systems. In the future, LLM's medical knowledge can be also used to improve the performance of vision-based medical-image CAD models.