Vision-Language Enhanced Foundation Model for Semi-supervised Medical Image Segmentation

TL;DR

This paper introduces VESSA, a vision-language enhanced foundation model that improves semi-supervised medical image segmentation by leveraging visual-semantic understanding and iterative pseudo-label refinement, significantly boosting accuracy with limited annotations.

Contribution

The work presents a novel VLM-based segmentation foundation model integrated into SSL, enabling effective semantic feature matching and iterative pseudo-label refinement for medical image segmentation.

Findings

VESSA outperforms existing methods on multiple datasets.

Significant accuracy improvements under limited annotations.

Effective integration of vision-language models into SSL frameworks.

Abstract

Semi-supervised learning (SSL) has emerged as an effective paradigm for medical image segmentation, reducing the reliance on extensive expert annotations. Meanwhile, vision-language models (VLMs) have demonstrated strong generalization and few-shot capabilities across diverse visual domains. In this work, we integrate VLM-based segmentation into semi-supervised medical image segmentation by introducing a Vision-Language Enhanced Semi-supervised Segmentation Assistant (VESSA) that incorporates foundation-level visual-semantic understanding into SSL frameworks. Our approach consists of two stages. In Stage 1, the VLM-enhanced segmentation foundation model VESSA is trained as a reference-guided segmentation assistant using a template bank containing gold-standard exemplars, simulating learning from limited labeled data. Given an input-template pair, VESSA performs visual feature matching to extract representative semantic and spatial cues from exemplar segmentations, generating structured prompts for a SAM2-inspired mask decoder to produce segmentation masks. In Stage 2, VESSA is integrated into a state-of-the-art SSL framework, enabling dynamic interaction with the student model: as student predictions become more refined, they are fed back to VESSA as prompts, allowing it to generate higher-quality pseudo-labels and stronger guidance. Extensive experiments across multiple segmentation datasets and domains show that VESSA-augmented SSL significantly enhances segmentation accuracy, outperforming state-of-the-art baselines under extremely limited annotation conditions.