MMLN: Leveraging Domain Knowledge for Multimodal Diagnosis

TL;DR

This paper introduces MMLN, a novel framework that integrates domain knowledge and multimodal data to improve lung disease diagnosis accuracy and interpretability, addressing limitations of existing model-free approaches.

Contribution

The paper presents a knowledge-driven, data-driven framework that formulates diagnosis rules from clinical guidelines and learns rule weights from text data for multimodal fusion.

Findings

Outperforms state-of-the-art multimodal baselines in accuracy

Enhances interpretability of diagnosis models

Effective use of domain knowledge reduces data dependence

Abstract

Recent studies show that deep learning models achieve good performance on medical imaging tasks such as diagnosis prediction. Among the models, multimodality has been an emerging trend, integrating different forms of data such as chest X-ray (CXR) images and electronic medical records (EMRs). However, most existing methods incorporate them in a model-free manner, which lacks theoretical support and ignores the intrinsic relations between different data sources. To address this problem, we propose a knowledge-driven and data-driven framework for lung disease diagnosis. By incorporating domain knowledge, machine learning models can reduce the dependence on labeled data and improve interpretability. We formulate diagnosis rules according to authoritative clinical medicine guidelines and learn the weights of rules from text data. Finally, a multimodal fusion consisting of text and image data is designed to infer the marginal probability of lung disease. We conduct experiments on a real-world dataset collected from a hospital. The results show that the proposed method outperforms the state-of-the-art multimodal baselines in terms of accuracy and interpretability.