Large Language Model Aided Birt-Hogg-Dube Syndrome Diagnosis with Multimodal Retrieval-Augmented Generation

TL;DR

This paper introduces BHD-RAG, a multimodal retrieval-augmented framework that enhances Birt-Hogg-Dube syndrome diagnosis from CT images by integrating domain-specific knowledge and clinical references to improve accuracy.

Contribution

The paper presents a novel multimodal retrieval-augmented approach that incorporates domain expertise and clinical data to improve diagnosis of rare lung diseases using large language models.

Findings

Achieved superior diagnostic accuracy on DCLD dataset.

Generated evidence-based descriptions aligned with expert insights.

Effectively integrated multimodal data for improved clinical diagnosis.

Abstract

Deep learning methods face dual challenges of limited clinical samples and low inter-class differentiation among Diffuse Cystic Lung Diseases (DCLDs) in advancing Birt-Hogg-Dube syndrome (BHD) diagnosis via Computed Tomography (CT) imaging. While Multimodal Large Language Models (MLLMs) demonstrate diagnostic potential fo such rare diseases, the absence of domain-specific knowledge and referable radiological features intensify hallucination risks. To address this problem, we propose BHD-RAG, a multimodal retrieval-augmented generation framework that integrates DCLD-specific expertise and clinical precedents with MLLMs to improve BHD diagnostic accuracy. BHDRAG employs: (1) a specialized agent generating imaging manifestation descriptions of CT images to construct a multimodal corpus of DCLDs cases. (2) a cosine similarity-based retriever pinpointing relevant imagedescription pairs for query images, and (3) an MLLM synthesizing retrieved evidence with imaging data for diagnosis. BHD-RAG is validated on the dataset involving four types of DCLDs, achieving superior accuracy and generating evidence-based descriptions closely aligned with expert insights.