# A multimodal artificial intelligence system for the detection and diagnosis of solid pancreatic lesions under EUS

## TL;DR

This paper introduces a multimodal AI system that improves the accuracy of diagnosing solid pancreatic lesions during endoscopic ultrasound.

## Contribution

The novel contribution is a deep learning-based multimodal AI system that integrates multiple data types for diagnosing pancreatic lesions.

## Key findings

- The AI system achieved 94.0% accuracy in differentiating carcinoma from noncancerous lesions.
- Multimodal models outperformed single-modality models with an AUC of 0.937.
- The AI system showed better diagnostic consistency and sensitivity than endoscopists.

## Abstract

Accurate differentiation of solid pancreatic lesions (SPLs) is crucial for treatment planning, but current methods still have limitations. Artificial intelligence (AI) has the potential to contribute to such diagnoses, yet existing AI models are restricted to focusing on a single modality. This study aims to develop a deep learning–based multimodal AI system to improve diagnostic accuracy for SPLs.

A retrospective analysis was conducted on 492 patients who underwent EUS for SPLs at Renmin Hospital of Wuhan University between December 2016 and September 2024. The AI system consisted of four deep learning models: DCNN1 for focal pancreatic lesion detection, DCNN2 for classifying pancreatic lesions as cystic or solid, DCNN3 for lesion boundary segmentation and size measurement, and DCNN4 for classifying carcinoma and noncancerous lesions. For DCNN4, four different modality models were constructed: (1) model A: EUS B-mode images only. (2) model B: EUS-E images only. (3) model C: EUS B-mode images and EUS-E images. and (4) model D: EUS B-mode images, EUS-E images, and clinical data. The model performance was compared with the diagnostic performance of endoscopists.

The accuracy values of DCNN1 and DCNN2 were 96.8% and 98.9%, respectively. The Dice coefficient of the DCNN3 was 0.876. Our AI system demonstrated high accuracy, sensitivity, and specificity in differentiating carcinoma from noncancerous SPLs. The multimodal models, particularly those integrating EUS B-mode and EUS-E images, outperformed single-modality models, achieving an accuracy of 94.0% and an AUC of 0.937. The AI model showed superior performance compared to endoscopists, with improved diagnostic consistency and sensitivity.

The multimodal AI system significantly improves the diagnostic accuracy of SPLs, providing a promising tool for the early detection and differentiation of pancreatic cancer.