Semi-MedRef: Semi-Supervised Medical Referring Image Segmentation with Cross-Modal Alignment

TL;DR

Semi-MedRef introduces a semi-supervised framework for medical image segmentation that maintains cross-modal alignment using novel augmentation and contrastive learning techniques, improving performance with limited labeled data.

Contribution

It proposes a new teacher-student SSL method with alignment-preserving components like T-PatchMix, PosAug, and ITCL for better image-text coherence in medical segmentation.

Findings

Outperforms fully supervised and semi-supervised baselines on QaTa-COV19 and MosMedData+ datasets.

Effectively maintains image-text alignment under strong augmentations.

Achieves consistent improvements across various label regimes.

Abstract

Medical referring image segmentation (MRIS) requires pixel-level masks aligned with textual descriptions of anatomical locations, making annotation costly in low-label regimes. Semi-supervised learning (SSL) can mitigate this burden by leveraging unlabeled data, but its success hinges on maintaining reliable image-text alignment under perturbations. Most existing SSL-based referred segmentation methods use either independent or simplistic multi-modal perturbations (e.g., left-right flips), without fully addressing cross-modal alignment under strong augmentation, while CutMix, highly effective in single-modal SSL, remains underexplored in multi-modal settings due to its tendency to disrupt image-text coherence. We propose Semi-MedRef, a teacher-student SSL framework designed to explicitly maintain consistency between medical images and positional language through three alignment-preserving components: T-PatchMix, a cross-modal CutMix-style augmentation that synchronizes patch mixing with referring expressions via position-constrained and probability-driven rules; PosAug, a position-aware text augmentation that masks or fuzzes anatomical phrases; and ITCL, a position-guided image-text contrastive learning module, which leverages positional pseudo-labels to construct soft anatomical positives and strengthen medically grounded cross-modal alignment. Experiments on QaTa-COV19 and MosMedData+ demonstrate that Semi-MedRef consistently outperforms both fully supervised and semi-supervised baselines across all label regimes.