Zero-shot System for Automatic Body Region Detection for Volumetric CT and MR Images

TL;DR

This paper explores zero-shot methods for automatic body region detection in volumetric CT and MR images using large pre-trained models, eliminating reliance on unreliable metadata and supervised training.

Contribution

It introduces and evaluates three training-free, zero-shot pipelines leveraging pre-trained models for anatomical region detection in medical images.

Findings

Segmentation-driven rule-based method achieves F1-scores of 0.947 (CT) and 0.914 (MR).

The rule-based approach demonstrates robustness across modalities and atypical scans.

MLLM methods perform well on visually distinctive regions but have limitations.

Abstract

Reliable identification of anatomical body regions is a prerequisite for many automated medical imaging workflows, yet existing solutions remain heavily dependent on unreliable DICOM metadata. Current solutions mainly use supervised learning, which limits their applicability in many real-world scenarios. In this work, we investigate whether body region detection in volumetric CT and MR images can be achieved in a fully zero-shot manner by using knowledge embedded in large pre-trained foundation models. We propose and systematically evaluate three training-free pipelines: (1) a segmentation-driven rule-based system leveraging pre-trained multi-organ segmentation models, (2) a Multimodal Large Language Model (MLLM) guided by radiologist-defined rules, and (3) a segmentation-aware MLLM that combines visual input with explicit anatomical evidence. All methods are evaluated on 887 heterogeneous CT and MR scans with manually verified anatomical region labels. The segmentation-driven rule-based approach achieves the strongest and most consistent performance, with weighted F1-scores of 0.947 (CT) and 0.914 (MR), demonstrating robustness across modalities and atypical scan coverage. The MLLM performs competitively in visually distinctive regions, while the segmentation-aware MLLM reveals fundamental limitations.