Boundary-Aware Instance Segmentation in Microscopy Imaging

TL;DR

This paper introduces a boundary-aware, prompt-free instance segmentation method for microscopy images that predicts signed distance functions to accurately separate touching cells, outperforming existing foundation-model approaches.

Contribution

It proposes a novel SDF-based segmentation framework with a unified MHD loss, enhancing boundary precision and instance separation without prompts.

Findings

Improved boundary accuracy over SAM-based methods

Robust separation of adjacent cell instances

Effective on both public and private datasets

Abstract

Accurate delineation of individual cells in microscopy videos is essential for studying cellular dynamics, yet separating touching or overlapping instances remains a persistent challenge. Although foundation-model for segmentation such as SAM have broadened the accessibility of image segmentation, they still struggle to separate nearby cell instances in dense microscopy scenes without extensive prompting. We propose a prompt-free, boundary-aware instance segmentation framework that predicts signed distance functions (SDFs) instead of binary masks, enabling smooth and geometry-consistent modeling of cell contours. A learned sigmoid mapping converts SDFs into probability maps, yielding sharp boundary localization and robust separation of adjacent instances. Training is guided by a unified Modified Hausdorff Distance (MHD) loss that integrates region- and boundary-based terms. Evaluations on both public and private high-throughput microscopy datasets demonstrate improved boundary accuracy and instance-level performance compared to recent SAM-based and foundation-model approaches. Source code is available at: https://github.com/ThomasMendelson/BAISeg.git