Training-free CryoET Tomogram Segmentation

TL;DR

CryoSAM is a novel training-free framework leveraging 2D foundation models for efficient, prompt-based 3D segmentation of cryo-electron tomograms, significantly reducing manual annotation efforts.

Contribution

The paper introduces CryoSAM, a training-free, prompt-based 3D segmentation method that automatically searches for features, enabling full tomogram segmentation with minimal prompts and annotations.

Findings

Outperforms existing methods significantly.

Requires fewer annotations for particle picking.

Effective in full tomogram segmentation of subcellular structures.

Abstract

Cryogenic Electron Tomography (CryoET) is a useful imaging technology in structural biology that is hindered by its need for manual annotations, especially in particle picking. Recent works have endeavored to remedy this issue with few-shot learning or contrastive learning techniques. However, supervised training is still inevitable for them. We instead choose to leverage the power of existing 2D foundation models and present a novel, training-free framework, CryoSAM. In addition to prompt-based single-particle instance segmentation, our approach can automatically search for similar features, facilitating full tomogram semantic segmentation with only one prompt. CryoSAM is composed of two major parts: 1) a prompt-based 3D segmentation system that uses prompts to complete single-particle instance segmentation recursively with Cross-Plane Self-Prompting, and 2) a Hierarchical Feature Matching mechanism that efficiently matches relevant features with extracted tomogram features. They collaborate to enable the segmentation of all particles of one category with just one particle-specific prompt. Our experiments show that CryoSAM outperforms existing works by a significant margin and requires even fewer annotations in particle picking. Further visualizations demonstrate its ability when dealing with full tomogram segmentation for various subcellular structures. Our code is available at: https://github.com/xulabs/aitom