LSP-DETR: Efficient and Scalable Nuclei Segmentation in Whole Slide Images

TL;DR

LSP-DETR is a novel, end-to-end transformer-based framework that efficiently segments nuclei in large whole slide images by representing nuclei as star-convex polygons, achieving high accuracy and speed without extensive post-processing.

Contribution

It introduces a lightweight transformer model with a new radial distance loss for natural overlapping nuclei segmentation, enabling scalable processing of gigapixel images.

Findings

Over five times faster than previous methods

Strong generalization across different tissue types

State-of-the-art efficiency and accuracy

Abstract

Precise and scalable instance segmentation of cell nuclei is essential for computational pathology, yet gigapixel Whole-Slide Images pose major computational challenges. Existing approaches rely on patch-based processing and costly post-processing for instance separation, sacrificing context and efficiency. We introduce LSP-DETR (Local Star Polygon DEtection TRansformer), a fully end-to-end framework that uses a lightweight transformer with linear complexity to process substantially larger images without additional computational cost. Nuclei are represented as star-convex polygons, and a novel radial distance loss function allows the segmentation of overlapping nuclei to emerge naturally, without requiring explicit overlap annotations or handcrafted post-processing. Evaluations on PanNuke and MoNuSeg show strong generalization across tissues and state-of-the-art efficiency, with LSP-DETR being over five times faster than the next-fastest leading method. Code and models are available at https://github.com/RationAI/lsp-detr.