Radiomics-Guided Global-Local Transformer for Weakly Supervised Pathology Localization in Chest X-Rays

TL;DR

This paper introduces a Radiomics-Guided Transformer that combines global image features with local radiomics information to improve weakly supervised pathology localization and classification in chest X-rays without requiring bounding box annotations.

Contribution

It proposes a novel end-to-end Radiomics-Guided Transformer architecture that fuses image and radiomic features for accurate disease localization and classification using only image-level labels.

Findings

Outperforms prior weakly supervised localization methods by 3.6% in IoU.

Improves disease classification accuracy by 1.1% AUC.

Demonstrates effective integration of radiomics and deep learning for medical imaging.

Abstract

Before the recent success of deep learning methods for automated medical image analysis, practitioners used handcrafted radiomic features to quantitatively describe local patches of medical images. However, extracting discriminative radiomic features relies on accurate pathology localization, which is difficult to acquire in real-world settings. Despite advances in disease classification and localization from chest X-rays, many approaches fail to incorporate clinically-informed domain knowledge. For these reasons, we propose a Radiomics-Guided Transformer (RGT) that fuses \textit{global} image information with \textit{local} knowledge-guided radiomics information to provide accurate cardiopulmonary pathology localization and classification \textit{without any bounding box annotations}. RGT consists of an image Transformer branch, a radiomics Transformer branch, and fusion layers that aggregate image and radiomic information. Using the learned self-attention of its image branch, RGT extracts a bounding box for which to compute radiomic features, which are further processed by the radiomics branch; learned image and radiomic features are then fused and mutually interact via cross-attention layers. Thus, RGT utilizes a novel end-to-end feedback loop that can bootstrap accurate pathology localization only using image-level disease labels. Experiments on the NIH ChestXRay dataset demonstrate that RGT outperforms prior works in weakly supervised disease localization (by an average margin of 3.6\% over various intersection-over-union thresholds) and classification (by 1.1\% in average area under the receiver operating characteristic curve). We publicly release our codes and pre-trained models at \url{https://github.com/VITA-Group/chext}.