GazeGNN: A Gaze-Guided Graph Neural Network for Chest X-ray Classification

TL;DR

GazeGNN introduces a real-time, gaze-guided graph neural network that directly utilizes raw eye-gaze data for chest X-ray classification, eliminating the need for preprocessing into visual attention maps and enhancing practical clinical application.

Contribution

It is the first GNN-based method to integrate raw eye-gaze data with images for disease classification, enabling real-time analysis without preprocessing.

Findings

Achieved superior classification performance on chest X-ray dataset.

Demonstrated real-time, end-to-end disease classification in clinical settings.

First to use GNN for direct integration of gaze and image data in medical imaging.

Abstract

Eye tracking research is important in computer vision because it can help us understand how humans interact with the visual world. Specifically for high-risk applications, such as in medical imaging, eye tracking can help us to comprehend how radiologists and other medical professionals search, analyze, and interpret images for diagnostic and clinical purposes. Hence, the application of eye tracking techniques in disease classification has become increasingly popular in recent years. Contemporary works usually transform gaze information collected by eye tracking devices into visual attention maps (VAMs) to supervise the learning process. However, this is a time-consuming preprocessing step, which stops us from applying eye tracking to radiologists' daily work. To solve this problem, we propose a novel gaze-guided graph neural network (GNN), GazeGNN, to leverage raw eye-gaze data without being converted into VAMs. In GazeGNN, to directly integrate eye gaze into image classification, we create a unified representation graph that models both images and gaze pattern information. With this benefit, we develop a real-time, real-world, end-to-end disease classification algorithm for the first time in the literature. This achievement demonstrates the practicality and feasibility of integrating real-time eye tracking techniques into the daily work of radiologists. To our best knowledge, GazeGNN is the first work that adopts GNN to integrate image and eye-gaze data. Our experiments on the public chest X-ray dataset show that our proposed method exhibits the best classification performance compared to existing methods. The code is available at https://github.com/ukaukaaaa/GazeGNN.