PAM: A Propagation-Based Model for Segmenting Any 3D Objects across Multi-Modal Medical Images

TL;DR

PAM is a versatile 3D medical image segmentation model that uses a 2D prompt and models slice relationships, outperforming existing methods in accuracy, speed, and generalizability across diverse datasets.

Contribution

This paper introduces PAM, a novel propagation-based segmentation model that effectively handles various 3D medical structures and imaging modalities with minimal manual input.

Findings

Outperformed existing models like MedSAM and SegVol by over 18.1% in DSC.

Demonstrated stable performance across different prompts and setups.

Reduced interaction time by approximately 63.6% with one-view prompts.

Abstract

Volumetric segmentation is important in medical imaging, but current methods face challenges like requiring lots of manual annotations and being tailored to specific tasks, which limits their versatility. General segmentation models used for natural images don't perform well with the unique features of medical images. There's a strong need for an adaptable approach that can effectively handle different 3D medical structures and imaging modalities. In this study, we present PAM (Propagating Anything Model), a segmentation approach that uses a 2D prompt, like a bounding box or sketch, to create a complete 3D segmentation of medical image volumes. PAM works by modeling relationships between slices, maintaining information flow across the 3D structure. It combines a CNN-based UNet for processing within slices and a Transformer-based attention module for propagating information between slices, leading to better generalizability across various imaging modalities. PAM significantly outperformed existing models like MedSAM and SegVol, with an average improvement of over 18.1% in dice similarity coefficient (DSC) across 44 medical datasets and various object types. It also showed stable performance despite prompt deviations and different propagation setups, and faster inference speeds compared to other models. PAM's one-view prompt design made it more efficient, reducing interaction time by about 63.6% compared to two-view prompts. Thanks to its focus on structural relationships, PAM handled unseen and complex objects well, showing a unique ability to generalize to new situations. PAM represents an advancement in medical image segmentation, effectively reducing the need for extensive manual work and specialized training. Its adaptability makes it a promising tool for more automated and reliable analysis in clinical settings.