MMGPL: Multimodal Medical Data Analysis with Graph Prompt Learning

TL;DR

This paper introduces a graph prompt learning approach for multimodal medical data analysis, focusing on neurological disorder diagnosis by emphasizing relevant brain patches and incorporating structural brain network information.

Contribution

The novel method learns graph prompts during fine-tuning, leveraging GPT-4 for concept relevance and GCNs for structural information, improving diagnosis accuracy over existing methods.

Findings

Achieves superior diagnostic performance compared to state-of-the-art methods.

Effectively reduces influence of irrelevant patches in neuroimaging data.

Validated by clinicians with extensive experiments.

Abstract

Prompt learning has demonstrated impressive efficacy in the fine-tuning of multimodal large models to a wide range of downstream tasks. Nonetheless, applying existing prompt learning methods for the diagnosis of neurological disorder still suffers from two issues: (i) existing methods typically treat all patches equally, despite the fact that only a small number of patches in neuroimaging are relevant to the disease, and (ii) they ignore the structural information inherent in the brain connection network which is crucial for understanding and diagnosing neurological disorders. To tackle these issues, we introduce a novel prompt learning model by learning graph prompts during the fine-tuning process of multimodal large models for diagnosing neurological disorders. Specifically, we first leverage GPT-4 to obtain relevant disease concepts and compute semantic similarity between these concepts and all patches. Secondly, we reduce the weight of irrelevant patches according to the semantic similarity between each patch and disease-related concepts. Moreover, we construct a graph among tokens based on these concepts and employ a graph convolutional network layer to extract the structural information of the graph, which is used to prompt the pre-trained multimodal large models for diagnosing neurological disorders. Extensive experiments demonstrate that our method achieves superior performance for neurological disorder diagnosis compared with state-of-the-art methods and validated by clinicians.