Organ-Aware Attention Improves CT Triage and Classification

TL;DR

This paper introduces ORACLE-CT, an organ-aware attention model that significantly improves CT triage and classification accuracy by leveraging localized evidence and organ-specific features, surpassing existing models on large public datasets.

Contribution

The study develops ORACLE-CT, a novel organ-aware attention mechanism that enhances CT classification performance and provides localized evidence, outperforming prior vision-language models and baselines.

Findings

Achieved AUROC 0.86 on CT-RATE for chest CTs.

Surpassed zero-shot VLM baseline on MERLIN abdomen dataset.

Delivered state-of-the-art results across chest and abdomen CT classification.

Abstract

There is an urgent need for triage and classification of high-volume medical imaging modalities such as computed tomography (CT), which can improve patient care and mitigate radiologist burnout. Study-level CT triage requires calibrated predictions with localized evidence; however, off-the-shelf Vision Language Models (VLM) struggle with 3D anatomy, protocol shifts, and noisy report supervision. This study used the two largest publicly available chest CT datasets: CT-RATE and RADCHEST-CT (held-out external test set). Our carefully tuned supervised baseline (instantiated as a simple Global Average Pooling head) establishes a new supervised state of the art, surpassing all reported linear-probe VLMs. Building on this baseline, we present ORACLE-CT, an encoder-agnostic, organ-aware head that pairs Organ-Masked Attention (mask-restricted, per-organ pooling that yields spatial evidence) with Organ-Scalar Fusion (lightweight fusion of normalized volume and mean-HU cues). In the chest setting, ORACLE-CT masked attention model achieves AUROC 0.86 on CT-RATE; in the abdomen setting, on MERLIN (30 findings), our supervised baseline exceeds a reproduced zero-shot VLM baseline obtained by running publicly released weights through our pipeline, and adding masked attention plus scalar fusion further improves performance to AUROC 0.85. Together, these results deliver state-of-the-art supervised classification performance across both chest and abdomen CT under a unified evaluation protocol. The source code is available at https://github.com/lavsendahal/oracle-ct.