A Generalist Foundation Model for Total-body PET/CT Enables Diagnostic Reporting and System-wide Metabolic Profiling

TL;DR

This paper introduces SDF-HOLO, a comprehensive multimodal foundation model for total-body PET/CT that enhances diagnostic accuracy, systemic metabolic profiling, and clinical report generation by integrating anatomical and metabolic information across the entire body.

Contribution

The paper presents SDF-HOLO, a novel dual-stream multimodal model pre-trained on over 10,000 patients, capable of holistic total-body PET/CT analysis with improved interpretability and diagnostic performance.

Findings

Outperforms existing models in tumor segmentation and lesion detection.

Reduces localization errors and hallucinated findings.

Enables system-wide metabolic profiling and inter-organ interaction analysis.

Abstract

Total-body PET/CT enables system-wide molecular imaging, but heterogeneous anatomical and metabolic signals, approximately 2 m axial coverage, and structured radiology semantics challenge existing medical AI models that assume single-modality inputs, localized fields of view, and coarse image-text alignment. We introduce SDF-HOLO (Systemic Dual-stream Fusion Holo Model), a multimodal foundation model for holistic total-body PET/CT, pre-trained on more than 10,000 patients. SDF-HOLO decouples CT and PET representation learning with dual-stream encoders and couples them through a cross-modal interaction module, allowing anatomical context to refine PET aggregation while metabolic saliency guides subtle morphological reasoning. To model long-range dependencies across the body, hierarchical context modeling combines efficient local windows with global attention. To bridge voxels and clinical language, we use anatomical segmentation masks as explicit semantic anchors and perform voxel-mask-text alignment during pre-training. Across tumor segmentation, low-dose lesion detection, and multilingual diagnostic report generation, SDF-HOLO outperforms strong task-specific and clinical-reference baselines while reducing localization errors and hallucinated findings. Beyond focal interpretation, the model enables system-wide metabolic profiling and reveals tumor-associated fingerprints of inter-organ metabolic network interactions, providing a scalable computational foundation for total-body PET/CT diagnostics and system-level precision oncology.