FGNet: Leveraging Feature-Guided Attention to Refine SAM2 for 3D EM Neuron Segmentation

TL;DR

This paper introduces FGNet, a novel framework that leverages pre-trained natural image models and a feature-guided attention mechanism to improve 3D neuron segmentation in Electron Microscopy images, addressing domain gaps and challenging morphologies.

Contribution

The paper proposes a new method that transfers knowledge from SAM2 to EM segmentation, using a feature-guided attention module and dual-affinity decoder for improved accuracy.

Findings

Achieves performance comparable to SOTA with frozen SAM2 weights.

Significantly outperforms SOTA after fine-tuning on EM data.

Validates the effectiveness of natural image pre-training for EM neuron segmentation.

Abstract

Accurate segmentation of neural structures in Electron Microscopy (EM) images is paramount for neuroscience. However, this task is challenged by intricate morphologies, low signal-to-noise ratios, and scarce annotations, limiting the accuracy and generalization of existing methods. To address these challenges, we seek to leverage the priors learned by visual foundation models on a vast amount of natural images to better tackle this task. Specifically, we propose a novel framework that can effectively transfer knowledge from Segment Anything 2 (SAM2), which is pre-trained on natural images, to the EM domain. We first use SAM2 to extract powerful, general-purpose features. To bridge the domain gap, we introduce a Feature-Guided Attention module that leverages semantic cues from SAM2 to guide a lightweight encoder, the Fine-Grained Encoder (FGE), in focusing on these challenging regions. Finally, a dual-affinity decoder generates both coarse and refined affinity maps. Experimental results demonstrate that our method achieves performance comparable to state-of-the-art (SOTA) approaches with the SAM2 weights frozen. Upon further fine-tuning on EM data, our method significantly outperforms existing SOTA methods. This study validates that transferring representations pre-trained on natural images, when combined with targeted domain-adaptive guidance, can effectively address the specific challenges in neuron segmentation.