FixationFormer: Direct Utilization of Expert Gaze Trajectories for Chest X-Ray Classification

TL;DR

FixationFormer leverages expert gaze trajectories as sequential tokens within a transformer architecture to improve chest X-ray classification accuracy by directly integrating diagnostic gaze data.

Contribution

This work introduces a novel transformer-based model that directly encodes expert gaze sequences for enhanced medical image analysis, addressing limitations of previous heatmap-based methods.

Findings

Achieves state-of-the-art performance on three chest X-ray datasets.

Demonstrates effective modeling of gaze data as sequences for better diagnostic cues.

Shows improved interpretability through explicit gaze-image attention mechanisms.

Abstract

Expert eye movements provide a rich, passive source of domain knowledge in radiology, offering a powerful cue for integrating diagnostic reasoning into computer-aided analysis. However, direct integration into CNN-based systems, which historically have dominated the medical image analysis domain, is challenging: gaze recordings are sequential, temporally dense yet spatially sparse, noisy, and variable across experts. As a consequence, most existing image-based models utilize reduced representations such as heatmaps. In contrast, gaze naturally aligns with transformer architectures, as both are sequential in nature and rely on attention to highlight relevant input regions. In this work, we introduce FixationFormer, a transformer-based architecture that represents expert gaze trajectories as sequences of tokens, thereby preserving their temporal and spatial structure. By modeling gaze sequences jointly with image features, our approach addresses sparsity and variability in gaze data while enabling a more direct and fine-grained integration of expert diagnostic cues through explicit cross-attention between the image and gaze token sequences. We evaluate our method on three publicly available benchmark chest X-ray datasets and demonstrate that it achieves state-of-the-art classification performance, highlighting the value of representing gaze as a sequence in transformer-based medical image analysis.