Rethinking Whole-Body CT Image Interpretation: An Abnormality-Centric Approach

TL;DR

This paper introduces a comprehensive approach for automated whole-body CT image interpretation focusing on abnormality detection and description, leveraging a new taxonomy, a large annotated dataset, and a novel model, OmniAbnorm-CT.

Contribution

It presents a new hierarchical abnormality taxonomy, a large annotated dataset, and a multi-purpose model for automatic abnormality grounding and description in CT images.

Findings

OmniAbnorm-CT outperforms existing methods in multiple validation scenarios.

The dataset includes over 14.5K CT images and 19K abnormalities with detailed annotations.

The taxonomy covers 404 abnormal findings across all body regions.

Abstract

Automated interpretation of CT images-particularly localizing and describing abnormal findings across multi-plane and whole-body scans-remains a significant challenge in clinical radiology. This work aims to address this challenge through four key contributions: (i) On taxonomy, we collaborate with senior radiologists to propose a comprehensive hierarchical classification system, with 404 representative abnormal findings across all body regions; (ii) On data, we contribute a dataset containing over 14.5K CT images from multiple planes and all human body regions, and meticulously provide grounding annotations for over 19K abnormalities, each linked to the detailed description and cast into the taxonomy; (iii) On model development, we propose OmniAbnorm-CT, which can automatically ground and describe abnormal findings on multi-plane and whole-body CT images based on text queries, while also allowing flexible interaction through visual prompts; (iv) On evaluation, we establish three representative tasks based on real clinical scenarios, and introduce a clinically grounded metric to assess abnormality descriptions. Through extensive experiments, we show that OmniAbnorm-CT can significantly outperform existing methods in both internal and external validations, and across all the tasks.