Advancing Generalizable Tumor Segmentation with Anomaly-Aware Open-Vocabulary Attention Maps and Frozen Foundation Diffusion Models

TL;DR

This paper introduces DiffuGTS, a novel framework leveraging frozen diffusion models to achieve zero-shot, open-vocabulary tumor segmentation with anomaly-aware attention maps, improving generalization and segmentation quality across diverse datasets.

Contribution

The paper proposes a new method using frozen foundation diffusion models and anomaly-aware attention maps for zero-shot tumor segmentation, enhancing generalization and segmentation quality.

Findings

Outperforms state-of-the-art models on multiple datasets

Achieves high-quality pseudo-healthy image generation

Enables open-vocabulary, zero-shot tumor segmentation

Abstract

We explore Generalizable Tumor Segmentation, aiming to train a single model for zero-shot tumor segmentation across diverse anatomical regions. Existing methods face limitations related to segmentation quality, scalability, and the range of applicable imaging modalities. In this paper, we uncover the potential of the internal representations within frozen medical foundation diffusion models as highly efficient zero-shot learners for tumor segmentation by introducing a novel framework named DiffuGTS. DiffuGTS creates anomaly-aware open-vocabulary attention maps based on text prompts to enable generalizable anomaly segmentation without being restricted by a predefined training category list. To further improve and refine anomaly segmentation masks, DiffuGTS leverages the diffusion model, transforming pathological regions into high-quality pseudo-healthy counterparts through latent space inpainting, and applies a novel pixel-level and feature-level residual learning approach, resulting in segmentation masks with significantly enhanced quality and generalization. Comprehensive experiments on four datasets and seven tumor categories demonstrate the superior performance of our method, surpassing current state-of-the-art models across multiple zero-shot settings. Codes are available at https://github.com/Yankai96/DiffuGTS.