Medical Multimodal Classifiers Under Scarce Data Condition

TL;DR

This paper presents a multimodal deep learning approach for medical image and report classification under scarce data conditions, incorporating novel anomaly detection and transfer learning to improve accuracy.

Contribution

It introduces a transfer learning-based multimodal classifier with a new anomaly detection technique that extends integrated gradients with unsupervised clustering.

Findings

Improves classification accuracy by 4-7% over individual models.

Uses transfer learning from large datasets and hospital-specific corpora.

Employs a novel anomaly detection method with a tunable parameter.

Abstract

Data is one of the essential ingredients to power deep learning research. Small datasets, especially specific to medical institutes, bring challenges to deep learning training stage. This work aims to develop a practical deep multimodal that can classify patients into abnormal and normal categories accurately as well as assist radiologists to detect visual and textual anomalies by locating areas of interest. The detection of the anomalies is achieved through a novel technique which extends the integrated gradients methodology with an unsupervised clustering algorithm. This technique also introduces a tuning parameter which trades off true positive signals to denoise false positive signals in the detection process. To overcome the challenges of the small training dataset which only has 3K frontal X-ray images and medical reports in pairs, we have adopted transfer learning for the multimodal which concatenates the layers of image and text submodels. The image submodel was trained on the vast ChestX-ray14 dataset, while the text submodel transferred a pertained word embedding layer from a hospital-specific corpus. Experimental results show that our multimodal improves the accuracy of the classification by 4% and 7% on average of 50 epochs, compared to the individual text and image model, respectively.