GEM: Context-Aware Gaze EstiMation with Visual Search Behavior Matching for Chest Radiograph

TL;DR

This paper introduces GEM, a novel context-aware gaze estimation model that simulates radiologists' visual search behavior in chest radiographs by integrating multimodal data and graph-based behavior matching, improving interpretability and accuracy.

Contribution

GEM is the first to incorporate context-awareness and visual behavior graph matching for more accurate and realistic gaze estimation in medical radiology interpretation.

Findings

GEM outperforms existing gaze estimation methods on four datasets.

The model demonstrates strong generalizability across different datasets.

GEM enhances interpretability of medical image analysis models.

Abstract

Gaze estimation is pivotal in human scene comprehension tasks, particularly in medical diagnostic analysis. Eye-tracking technology facilitates the recording of physicians' ocular movements during image interpretation, thereby elucidating their visual attention patterns and information-processing strategies. In this paper, we initially define the context-aware gaze estimation problem in medical radiology report settings. To understand the attention allocation and cognitive behavior of radiologists during the medical image interpretation process, we propose a context-aware Gaze EstiMation (GEM) network that utilizes eye gaze data collected from radiologists to simulate their visual search behavior patterns throughout the image interpretation process. It consists of a context-awareness module, visual behavior graph construction, and visual behavior matching. Within the context-awareness module, we achieve intricate multimodal registration by establishing connections between medical reports and images. Subsequently, for a more accurate simulation of genuine visual search behavior patterns, we introduce a visual behavior graph structure, capturing such behavior through high-order relationships (edges) between gaze points (nodes). To maintain the authenticity of visual behavior, we devise a visual behavior-matching approach, adjusting the high-order relationships between them by matching the graph constructed from real and estimated gaze points. Extensive experiments on four publicly available datasets demonstrate the superiority of GEM over existing methods and its strong generalizability, which also provides a new direction for the effective utilization of diverse modalities in medical image interpretation and enhances the interpretability of models in the field of medical imaging. https://github.com/Tiger-SN/GEM