Using Multi-modal Data for Improving Generalizability and Explainability of Disease Classification in Radiology

TL;DR

This study investigates how combining radiology images, reports, and eye-gaze data affects deep learning models' accuracy and explainability in disease classification, finding that eye-gaze data enhances interpretability but not performance.

Contribution

It provides a comprehensive analysis of the impact of eye-gaze data on model performance and explainability in radiology classification tasks.

Findings

Best performance with image and report data

Eye-gaze data improves model explainability

Eye-gaze data does not boost classification accuracy

Abstract

Traditional datasets for the radiological diagnosis tend to only provide the radiology image alongside the radiology report. However, radiology reading as performed by radiologists is a complex process, and information such as the radiologist's eye-fixations over the course of the reading has the potential to be an invaluable data source to learn from. Nonetheless, the collection of such data is expensive and time-consuming. This leads to the question of whether such data is worth the investment to collect. This paper utilizes the recently published Eye-Gaze dataset to perform an exhaustive study on the impact on performance and explainability of deep learning (DL) classification in the face of varying levels of input features, namely: radiology images, radiology report text, and radiologist eye-gaze data. We find that the best classification performance of X-ray images is achieved with a combination of radiology report free-text and radiology image, with the eye-gaze data providing no performance boost. Nonetheless, eye-gaze data serving as secondary ground truth alongside the class label results in highly explainable models that generate better attention maps compared to models trained to do classification and attention map generation without eye-gaze data.