A novel attention mechanism for noise-adaptive and robust segmentation of microtubules in microscopy images

TL;DR

This paper introduces ASE_Res_UNet, a lightweight, noise-adaptive attention-based model that significantly improves microtubule segmentation in noisy microscopy images, outperforming existing methods and demonstrating strong transferability.

Contribution

The paper proposes a novel noise-adaptive attention mechanism integrated into a U-Net architecture, along with a synthetic dataset generation strategy, to enhance segmentation robustness under challenging conditions.

Findings

ASE_Res_UNet outperforms ablated variants and alternative models in noisy synthetic images.

It achieves superior segmentation accuracy on real microscopy datasets.

The model demonstrates strong transferability to other curvilinear structures like blood vessels and nerves.

Abstract

Segmenting cytoskeletal filaments in microscopy images is essential for studying their roles in cellular processes. However, this task is highly challenging due to the fine, densely packed, and intertwined nature of these structures. Imaging limitations further complicate analysis. While deep learning has advanced segmentation of large, well-defined biological structures, its performance often degrades under such adverse conditions. Additional challenges include obtaining precise annotations for curvilinear structures and managing severe class imbalance during training. We introduce a novel noise-adaptive attention mechanism that extends the Squeeze-and-Excitation (SE) module to dynamically adjust to varying noise levels. Integrated into a U-Net decoder with residual encoder blocks, this yields ASE_Res_UNet, a lightweight yet high-performance model. We also developed a synthetic dataset generation strategy that ensures accurate annotations of fine filaments in noisy images. We systematically evaluated loss functions and metrics to mitigate class imbalance, ensuring robust performance assessment. ASE_Res_UNet effectively segmented microtubules in noisy synthetic images, outperforming its ablated variants. It also demonstrated superior segmentation compared to models with alternative attention mechanisms or distinct architectures, while requiring fewer parameters, making it efficient for resource-constrained environments. Evaluation on a newly curated real microscopy dataset and a recently reannotated dataset highlighted ASE_Res_UNet's effectiveness in segmenting microtubules beyond synthetic images. For these datasets, ASE_Res_UNet was competitive with a recent synthetic data-driven approach that shares two cytoskeleton pretrained models. Importantly, ASE_Res_UNet showed strong transferability to other curvilinear structures (blood vessels and nerves) across diverse imaging conditions.