Eyes on the Image: Gaze Supervised Multimodal Learning for Chest X-ray Diagnosis and Report Generation

TL;DR

This paper introduces a two-stage multimodal framework that uses gaze data to improve disease classification and report generation from chest X-rays, enhancing interpretability and diagnostic accuracy.

Contribution

It presents a novel gaze-guided contrastive learning architecture and a modular report generation pipeline leveraging eye-tracking data for better medical image analysis.

Findings

Gaze-informed models outperform baseline in classification metrics.

Incorporating eye-tracking data improves report relevance and accuracy.

The approach enhances interpretability of AI in radiology.

Abstract

We propose a two-stage multimodal framework that enhances disease classification and region-aware radiology report generation from chest X-rays, leveraging the MIMIC-Eye dataset. In the first stage, we introduce a gaze-guided contrastive learning architecture for disease classification. It integrates visual features, clinical labels, bounding boxes, and radiologist eye-tracking signals and is equipped with a novel multi-term gaze-attention loss combining MSE, KL divergence, correlation, and center-of-mass alignment. Incorporating fixations improves F1 score from 0.597 to 0.631 (+5.70%) and AUC from 0.821 to 0.849 (+3.41%), while also improving precision and recall, highlighting the effectiveness of gaze-informed attention supervision. In the second stage, we present a modular report generation pipeline that extracts confidence-weighted diagnostic keywords, maps them to anatomical regions using a curated dictionary constructed from domain-specific priors, and generates region-aligned sentences via structured prompts. This pipeline improves report quality as measured by clinical keyword recall and ROUGE overlap. Our results demonstrate that integrating gaze data improves both classification performance and the interpretability of generated medical reports.