Gaze-directed Vision GNN for Mitigating Shortcut Learning in Medical Image

TL;DR

This paper introduces GD-ViG, a gaze-directed Vision GNN that uses radiologists' gaze patterns to focus on disease-relevant regions, reducing shortcut learning and improving interpretability in medical image analysis.

Contribution

The paper proposes a novel gaze-guided GNN framework that leverages radiologists' gaze data to mitigate shortcut learning without requiring real gaze data during inference.

Findings

GD-ViG outperforms state-of-the-art methods on medical image datasets.

It effectively mitigates shortcut learning and enhances interpretability.

The model does not need real gaze data during inference.

Abstract

Deep neural networks have demonstrated remarkable performance in medical image analysis. However, its susceptibility to spurious correlations due to shortcut learning raises concerns about network interpretability and reliability. Furthermore, shortcut learning is exacerbated in medical contexts where disease indicators are often subtle and sparse. In this paper, we propose a novel gaze-directed Vision GNN (called GD-ViG) to leverage the visual patterns of radiologists from gaze as expert knowledge, directing the network toward disease-relevant regions, and thereby mitigating shortcut learning. GD-ViG consists of a gaze map generator (GMG) and a gaze-directed classifier (GDC). Combining the global modelling ability of GNNs with the locality of CNNs, GMG generates the gaze map based on radiologists' visual patterns. Notably, it eliminates the need for real gaze data during inference, enhancing the network's practical applicability. Utilizing gaze as the expert knowledge, the GDC directs the construction of graph structures by incorporating both feature distances and gaze distances, enabling the network to focus on disease-relevant foregrounds. Thereby avoiding shortcut learning and improving the network's interpretability. The experiments on two public medical image datasets demonstrate that GD-ViG outperforms the state-of-the-art methods, and effectively mitigates shortcut learning. Our code is available at https://github.com/SX-SS/GD-ViG.