Multi-modal Fusion for Diabetes Mellitus and Impaired Glucose Regulation Detection

TL;DR

This paper introduces a multi-modal classification approach combining tongue, face, and sublingual images to improve the non-invasive diagnosis of Diabetes Mellitus and Impaired Glucose Regulation, demonstrating superior accuracy over single-modality methods.

Contribution

It proposes a novel multi-modal learning framework (MMSSL) that leverages complementary information from multiple non-invasive modalities for disease diagnosis.

Findings

Outperforms single-modality diagnosis methods.

Effective on a dataset of 192 healthy, 198 DM, and 114 IGR samples.

Validates the benefit of multi-modal fusion in medical diagnosis.

Abstract

Effective and accurate diagnosis of Diabetes Mellitus (DM), as well as its early stage Impaired Glucose Regulation (IGR), has attracted much attention recently. Traditional Chinese Medicine (TCM) [3], [5] etc. has proved that tongue, face and sublingual diagnosis as a noninvasive method is a reasonable way for disease detection. However, most previous works only focus on a single modality (tongue, face or sublingual) for diagnosis, although different modalities may provide complementary information for the diagnosis of DM and IGR. In this paper, we propose a novel multi-modal classification method to discriminate between DM (or IGR) and healthy controls. Specially, the tongue, facial and sublingual images are first collected by using a non-invasive capture device. The color, texture and geometry features of these three types of images are then extracted, respectively. Finally, our so-called multi-modal similar and specific learning (MMSSL) approach is proposed to combine features of tongue, face and sublingual, which not only exploits the correlation but also extracts individual components among them. Experimental results on a dataset consisting of 192 Healthy, 198 DM and 114 IGR samples (all samples were obtained from Guangdong Provincial Hospital of Traditional Chinese Medicine) substantiate the effectiveness and superiority of our proposed method for the diagnosis of DM and IGR, compared to the case of using a single modality.