Developing Foundation Models for Universal Segmentation from 3D Whole-Body Positron Emission Tomography

TL;DR

This paper introduces SegAnyPET, a universal 3D PET segmentation model trained on the largest PET dataset to date, enabling accurate, scalable, and efficient organ and lesion segmentation across diverse clinical scenarios.

Contribution

The paper presents a novel foundational model for PET segmentation, supported by the largest PET dataset, and demonstrates its strong zero-shot performance across multiple tasks.

Findings

SegAnyPET achieves high zero-shot accuracy on diverse PET segmentation tasks.

The model supports efficient human correction and clinical workflows.

Extensive multi-center evaluations validate its robustness and generalizability.

Abstract

Positron emission tomography (PET) is a key nuclear medicine imaging modality that visualizes radiotracer distributions to quantify in vivo physiological and metabolic processes, playing an irreplaceable role in disease management. Despite its clinical importance, the development of deep learning models for quantitative PET image analysis remains severely limited, driven by both the inherent segmentation challenge from PET's paucity of anatomical contrast and the high costs of data acquisition and annotation. To bridge this gap, we develop generalist foundational models for universal segmentation from 3D whole-body PET imaging. We first build the largest and most comprehensive PET dataset to date, comprising 11041 3D whole-body PET scans with 59831 segmentation masks for model development. Based on this dataset, we present SegAnyPET, an innovative foundational model with general-purpose applicability to diverse segmentation tasks. Built on a 3D architecture with a prompt engineering strategy for mask generation, SegAnyPET enables universal and scalable organ and lesion segmentation, supports efficient human correction with minimal effort, and enables a clinical human-in-the-loop workflow. Extensive evaluations on multi-center, multi-tracer, multi-disease datasets demonstrate that SegAnyPET achieves strong zero-shot performance across a wide range of segmentation tasks, highlighting its potential to advance the clinical applications of molecular imaging.