RadioTransformer: A Cascaded Global-Focal Transformer for Visual Attention-guided Disease Classification

TL;DR

RadioTransformer introduces a novel transformer framework that incorporates radiologists' gaze patterns to improve disease classification in chest radiographs, effectively leveraging domain-specific visual search behavior.

Contribution

It is the first to integrate radiologists' eye-gaze data into a cascaded global-focal transformer for medical image diagnosis, enhancing interpretability and accuracy.

Findings

Improved classification accuracy across 8 datasets.

Effective learning from gaze data with a student-teacher approach.

Robust performance without gaze data during inference.

Abstract

In this work, we present RadioTransformer, a novel visual attention-driven transformer framework, that leverages radiologists' gaze patterns and models their visuo-cognitive behavior for disease diagnosis on chest radiographs. Domain experts, such as radiologists, rely on visual information for medical image interpretation. On the other hand, deep neural networks have demonstrated significant promise in similar tasks even where visual interpretation is challenging. Eye-gaze tracking has been used to capture the viewing behavior of domain experts, lending insights into the complexity of visual search. However, deep learning frameworks, even those that rely on attention mechanisms, do not leverage this rich domain information. RadioTransformer fills this critical gap by learning from radiologists' visual search patterns, encoded as 'human visual attention regions' in a cascaded global-focal transformer framework. The overall 'global' image characteristics and the more detailed 'local' features are captured by the proposed global and focal modules, respectively. We experimentally validate the efficacy of our student-teacher approach for 8 datasets involving different disease classification tasks where eye-gaze data is not available during the inference phase. Code: https://github.com/bmi-imaginelab/radiotransformer.