Prompt-Free Lightweight SAM Adaptation for Histopathology Nuclei Segmentation with Strong Cross-Dataset Generalization

TL;DR

This paper introduces a prompt-free, lightweight adaptation of the Segment Anything Model (SAM) for histopathology nuclei segmentation, achieving high accuracy and strong cross-dataset generalization with minimal trainable parameters.

Contribution

It proposes a novel prompt-free, lightweight SAM adaptation that fine-tunes only LoRA modules, enabling efficient and generalizable nuclei segmentation in histopathology images.

Findings

Achieves state-of-the-art results on three benchmark datasets.

Requires only 4.1 million trainable parameters.

Demonstrates strong cross-dataset generalization.

Abstract

Histopathology nuclei segmentation is crucial for quantitative tissue analysis and cancer diagnosis. Although existing segmentation methods have achieved strong performance, they are often computationally heavy and show limited generalization across datasets, which constrains their practical deployment. Recent SAM-based approaches have shown great potential in general and medical imaging, but typically rely on prompt guidance or complex decoders, making them less suitable for histopathology images with dense nuclei and heterogeneous appearances. We propose a prompt-free and lightweight SAM adaptation that leverages multi-level encoder features and residual decoding for accurate and efficient nuclei segmentation. The framework fine-tunes only LoRA modules within the frozen SAM encoder, requiring just 4.1M trainable parameters. Experiments on three benchmark datasets TNBC, MoNuSeg, and PanNuke demonstrate state-of-the-art performance and strong cross-dataset generalization, highlighting the effectiveness and practicality of the proposed framework for histopathology applications.