Region-Normalized DPO for Medical Image Segmentation under Noisy Judges

TL;DR

This paper introduces Region-Normalized DPO, a novel method that improves medical image segmentation by stabilizing preference-based fine-tuning using noisy automatic quality signals, reducing harmful updates and enhancing performance.

Contribution

The paper proposes RN-DPO, a segmentation-aware normalization technique for DPO that mitigates noise bias in preference signals, leading to more stable and effective fine-tuning in medical image segmentation.

Findings

RN-DPO outperforms standard DPO in multiple medical datasets.

Normalization reduces harmful updates from noisy preferences.

RN-DPO stabilizes training and improves segmentation performance.

Abstract

While dense pixel-wise annotations remain the gold standard for medical image segmentation, they are costly to obtain and limit scalability. In contrast, many deployed systems already produce inexpensive automatic quality-control (QC) signals like model agreement, uncertainty measures, or learned mask-quality scores which can be used for further model training without additional ground-truth annotation. However, these signals can be noisy and biased, making preference-based fine-tuning susceptible to harmful updates. We study Direct Preference Optimization (DPO) for segmentation from such noisy judges using proposals generated by a supervised base segmenter trained on a small labeled set. We find that outcomes depend strongly on how preference pairs are mined: selecting the judge's top-ranked proposal can improve peak performance when the judge is reliable, but can amplify harmful errors under weaker judges. We propose Region-Normalized DPO (RN-DPO), a segmentation-aware objective which normalizes preference updates by the size of the disagreement region between masks, reducing the leverage of harmful comparisons and improving optimization stability. Across two medical datasets and multiple regimes, RN-DPO improves sustained performance and stabilizes preference-based fine-tuning, outperforming standard DPO and strong baselines without requiring additional pixel annotations.