SAMPO-Path: Segmentation Intent-Aligned Preference Optimization for Pathology Foundation Model Segmentation

TL;DR

SAMPO is a novel fine-tuning framework that aligns pathology foundation models with clinical segmentation intent, improving accuracy and robustness in histopathology image segmentation despite noisy prompts.

Contribution

It introduces the first adaptation of Direct Preference Optimization to vision models, enabling effective preference-based fine-tuning for pathology segmentation.

Findings

Consistently improves segmentation accuracy across multiple datasets.

Enhances robustness to prompt variations and noisy inputs.

Aligns model outputs more closely with clinical segmentation intent.

Abstract

Foundation models have shown strong performance in multi-object segmentation with visual prompts, yet histopathology images remain challenging due to high cellular density, heterogeneity, and the gap between pixel-level supervision and clinical segmentation intent (e.g., selectively segmenting nuclei of a specific type). In practice, such intents are expressed through diverse and noisy prompts, causing prompt-intent misalignment and inconsistent predictions. We introduce SAMPO (Segmentation Anything Model with Preference Optimization), a preference-aligned fine-tuning framework that explicitly aligns pathology foundation models with clinical segmentation intent. SAMPO is the first to adapt Direct Preference Optimization (DPO) to pure vision foundation models, enabling accurate segmentation from minimal and imperfect prompts. The framework features three key components: (1) online prompt-centric preference mining to synthesize preference pairs across prompt qualities; (2) multi-mask preference learning to leverage output ambiguity for fine-grained ranking supervision; and (3) a hybrid loss combining preference optimization with pixel-level supervision for stable training. Trained on two datasets covering four tasks and evaluated on corresponding test sets and 12 external validation datasets, SAMPO consistently improves segmentation accuracy, robustness to prompt variations, and clinical intent adherence in dense histopathology images.