An Open Multi-Center Whole-Body FDG PET/CT Foundation Model for Tumor Segmentation

TL;DR

This paper introduces an open-source, multi-center FDG PET/CT foundation model that leverages hierarchical UNet architectures and masked autoencoding to improve tumor segmentation efficiency and cross-modality learning.

Contribution

It presents a novel multi-center foundation model with early cross-modal interaction and a masked autoencoding objective, enhancing label efficiency and generalization in PET/CT tumor segmentation.

Findings

Achieves comparable performance with only 10% labeled data.

Outperforms separated-modality pretraining in 5-shot linear probing.

Demonstrates robust cross-modality representation learning.

Abstract

The synergistic interpretation of anatomical information from computed tomography (CT) and metabolic information from positron emission tomography (PET) is important to oncologic imaging. However, existing deep learning methods for PET/CT remain largely task-specific, are often trained on single-center cohorts, or adopt dual-branch fusion schemes that delay cross-modal interaction and underutilize early spatial correspondence between PET and CT. To address these limitations, we present an open-source, multi-center, whole-body FDG PET/CT foundation model utilizing 4,997 harmonized scans from four public datasets. Our framework employs hierarchical UNet-shaped backbones with early channel-wise concatenation, enabling anatomical and metabolic features to interact from the first embedding layer onward. We further introduce a masked autoencoding objective based on zero-mean imputation, combined with a weighted global reconstruction loss. This design avoids non-physical intensity discontinuities at masked-region boundaries that arise from learnable mask tokens. On downstream AutoPET lesion segmentation, the proposed models demonstrate strong label efficiency: with only 10\% of the labeled training data, they achieve performance comparable to models trained from scratch on the full dataset. Under extreme 5-shot linear probing, joint PET/CT pretraining also achieves higher Dice scores than separated-modality pretraining. This multi-center foundation model demonstrates label efficiency and cross-modality representation learning for PET/CT tumor segmentation. It provides a robust, open-source basis for advancing automated oncologic imaging, significantly reducing the need for large-scale manual annotations in clinical practice.