Multimodal Medical Disease Classification with LLaMA II

TL;DR

This paper retrains a multimodal transformer model using LLaMA II to classify medical diseases by effectively fusing text and image data, achieving high accuracy on chest X-ray datasets.

Contribution

It introduces a novel multimodal architecture with early fusion techniques using LLaMA II, outperforming previous models on medical multimodal classification tasks.

Findings

Early fusion yields better results than late fusion.

Best model achieves 97.10% mean AUC.

Model outperforms previous classification models.

Abstract

Medical patient data is always multimodal. Images, text, age, gender, histopathological data are only few examples for different modalities in this context. Processing and integrating this multimodal data with deep learning based methods is of utmost interest due to its huge potential for medical procedure such as diagnosis and patient treatment planning. In this work we retrain a multimodal transformer-based model for disease classification. To this end we use the text-image pair dataset from OpenI consisting of 2D chest X-rays associated with clinical reports. Our focus is on fusion methods for merging text and vision information extracted from medical datasets. Different architecture structures with a LLaMA II backbone model are tested. Early fusion of modality specific features creates better results with the best model reaching 97.10% mean AUC than late fusion from a deeper level of the architecture (best model: 96.67% mean AUC). Both outperform former classification models tested on the same multimodal dataset. The newly introduced multimodal architecture can be applied to other multimodal datasets with little effort and can be easily adapted for further research, especially, but not limited to, the field of medical AI.