Segment Any Tumour: An Uncertainty-Aware Vision Foundation Model for Whole-Body Analysis

TL;DR

This paper introduces SAT3D, a novel uncertainty-aware 3D vision foundation model for robust, generalisable tumour segmentation across diverse medical imaging modalities, addressing challenges in ambiguous and low-contrast regions.

Contribution

We develop SAT3D, integrating a shifted-window transformer and uncertainty-guided training, and demonstrate its superior performance over existing models on multiple datasets.

Findings

SAT3D outperforms recent foundation models and nnUNet in segmentation accuracy.

The model generalises well across different imaging modalities and tumour types.

Interactive 3D Slicer plugin facilitates clinical application.

Abstract

Prompt-driven vision foundation models, such as the Segment Anything Model, have recently demonstrated remarkable adaptability in computer vision. However, their direct application to medical imaging remains challenging due to heterogeneous tissue structures, imaging artefacts, and low-contrast boundaries, particularly in tumours and cancer primaries leading to suboptimal segmentation in ambiguous or overlapping lesion regions. Here, we present Segment Any Tumour 3D (SAT3D), a lightweight volumetric foundation model designed to enable robust and generalisable tumour segmentation across diverse medical imaging modalities. SAT3D integrates a shifted-window vision transformer for hierarchical volumetric representation with an uncertainty-aware training pipeline that explicitly incorporates uncertainty estimates as prompts to guide reliable boundary prediction in low-contrast regions. Adversarial learning further enhances model performance for the ambiguous pathological regions. We benchmark SAT3D against three recent vision foundation models and nnUNet across 11 publicly available datasets, encompassing 3,884 tumour and cancer cases for training and 694 cases for in-distribution evaluation. Trained on 17,075 3D volume-mask pairs across multiple modalities and cancer primaries, SAT3D demonstrates strong generalisation and robustness. To facilitate practical use and clinical translation, we developed a 3D Slicer plugin that enables interactive, prompt-driven segmentation and visualisation using the trained SAT3D model. Extensive experiments highlight its effectiveness in improving segmentation accuracy under challenging and out-of-distribution scenarios, underscoring its potential as a scalable foundation model for medical image analysis.