Eye-gaze-guided Vision Transformer for Rectifying Shortcut Learning

TL;DR

This paper introduces EG-ViT, a vision transformer guided by radiologists' eye-gaze data to improve medical image diagnosis, reduce shortcut learning, and enhance interpretability with limited data.

Contribution

The paper proposes a novel eye-gaze-guided vision transformer that incorporates expert visual attention to improve medical diagnosis and model interpretability.

Findings

EG-ViT outperforms baseline models on INbreast and SIIM-ACR datasets.

The model effectively rectifies shortcut learning and biases.

It demonstrates improved interpretability and generalization in medical imaging.

Abstract

Learning harmful shortcuts such as spurious correlations and biases prevents deep neural networks from learning the meaningful and useful representations, thus jeopardizing the generalizability and interpretability of the learned representation. The situation becomes even more serious in medical imaging, where the clinical data (e.g., MR images with pathology) are limited and scarce while the reliability, generalizability and transparency of the learned model are highly required. To address this problem, we propose to infuse human experts' intelligence and domain knowledge into the training of deep neural networks. The core idea is that we infuse the visual attention information from expert radiologists to proactively guide the deep model to focus on regions with potential pathology and avoid being trapped in learning harmful shortcuts. To do so, we propose a novel eye-gaze-guided vision transformer (EG-ViT) for diagnosis with limited medical image data. We mask the input image patches that are out of the radiologists' interest and add an additional residual connection in the last encoder layer of EG-ViT to maintain the correlations of all patches. The experiments on two public datasets of INbreast and SIIM-ACR demonstrate our EG-ViT model can effectively learn/transfer experts' domain knowledge and achieve much better performance than baselines. Meanwhile, it successfully rectifies the harmful shortcut learning and significantly improves the EG-ViT model's interpretability. In general, EG-ViT takes the advantages of both human expert's prior knowledge and the power of deep neural networks. This work opens new avenues for advancing current artificial intelligence paradigms by infusing human intelligence.