MDNet: A Semantically and Visually Interpretable Medical Image Diagnosis Network

TL;DR

MDNet is a novel medical diagnosis network that provides interpretable predictions by linking images and reports, enabling visualization, report generation, and retrieval, trained end-to-end on bladder cancer data.

Contribution

Introduces MDNet, a multimodal network with enhanced feature utilization and attention mechanisms for interpretable medical diagnosis and report generation.

Findings

MDNet outperforms baseline models on diagnostic accuracy.

The image model achieves state-of-the-art on CIFAR datasets.

The network provides visual and textual explanations for diagnoses.

Abstract

The inability to interpret the model prediction in semantically and visually meaningful ways is a well-known shortcoming of most existing computer-aided diagnosis methods. In this paper, we propose MDNet to establish a direct multimodal mapping between medical images and diagnostic reports that can read images, generate diagnostic reports, retrieve images by symptom descriptions, and visualize attention, to provide justifications of the network diagnosis process. MDNet includes an image model and a language model. The image model is proposed to enhance multi-scale feature ensembles and utilization efficiency. The language model, integrated with our improved attention mechanism, aims to read and explore discriminative image feature descriptions from reports to learn a direct mapping from sentence words to image pixels. The overall network is trained end-to-end by using our developed optimization strategy. Based on a pathology bladder cancer images and its diagnostic reports (BCIDR) dataset, we conduct sufficient experiments to demonstrate that MDNet outperforms comparative baselines. The proposed image model obtains state-of-the-art performance on two CIFAR datasets as well.