MedSeg-R: Medical Image Segmentation with Clinical Reasoning

TL;DR

MedSeg-R introduces a dual-stage framework inspired by clinical reasoning that enhances medical image segmentation by integrating semantic priors and dynamic feature modulation, significantly improving detection of small and ambiguous lesions.

Contribution

The paper presents MedSeg-R, a novel lightweight framework that incorporates structured semantic priors into the segmentation process, improving generalization and sensitivity over existing methods.

Findings

Achieves large Dice score improvements on challenging benchmarks.

Effectively detects small and overlapping lesions.

Demonstrates compatibility with SAM-based systems.

Abstract

Medical image segmentation is challenging due to overlapping anatomies with ambiguous boundaries and a severe imbalance between the foreground and background classes, which particularly affects the delineation of small lesions. Existing methods, including encoder-decoder networks and prompt-driven variants of the Segment Anything Model (SAM), rely heavily on local cues or user prompts and lack integrated semantic priors, thus failing to generalize well to low-contrast or overlapping targets. To address these issues, we propose MedSeg-R, a lightweight, dual-stage framework inspired by inspired by clinical reasoning. Its cognitive stage interprets medical report into structured semantic priors (location, texture, shape), which are fused via transformer block. In the perceptual stage, these priors modulate the SAM backbone: spatial attention highlights likely lesion regions, dynamic convolution adapts feature filters to expected textures, and deformable sampling refines spatial support. By embedding this fine-grained guidance early, MedSeg-R disentangles inter-class confusion and amplifies minority-class cues, greatly improving sensitivity to small lesions. In challenging benchmarks, MedSeg-R produces large Dice improvements in overlapping and ambiguous structures, demonstrating plug-and-play compatibility with SAM-based systems.