Alleviating Textual Reliance in Medical Language-guided Segmentation via Prototype-driven Semantic Approximation

TL;DR

ProLearn introduces a prototype-driven framework that reduces reliance on paired text-image data in medical segmentation, enabling effective segmentation even without textual guidance.

Contribution

The paper proposes a novel Prototype-driven Semantic Approximation module that allows language-guided segmentation models to operate with less or no textual input, addressing key limitations of existing methods.

Findings

ProLearn outperforms state-of-the-art methods with limited textual data.

The PSA module effectively approximates semantic guidance from textual reports.

ProLearn enables segmentation on image-only data, broadening clinical applicability.

Abstract

Medical language-guided segmentation, integrating textual clinical reports as auxiliary guidance to enhance image segmentation, has demonstrated significant improvements over unimodal approaches. However, its inherent reliance on paired image-text input, which we refer to as ``textual reliance", presents two fundamental limitations: 1) many medical segmentation datasets lack paired reports, leaving a substantial portion of image-only data underutilized for training; and 2) inference is limited to retrospective analysis of cases with paired reports, limiting its applicability in most clinical scenarios where segmentation typically precedes reporting. To address these limitations, we propose ProLearn, the first Prototype-driven Learning framework for language-guided segmentation that fundamentally alleviates textual reliance. At its core, we introduce a novel Prototype-driven Semantic Approximation (PSA) module to enable approximation of semantic guidance from textual input. PSA initializes a discrete and compact prototype space by distilling segmentation-relevant semantics from textual reports. Once initialized, it supports a query-and-respond mechanism which approximates semantic guidance for images without textual input, thereby alleviating textual reliance. Extensive experiments on QaTa-COV19, MosMedData+ and Kvasir-SEG demonstrate that ProLearn outperforms state-of-the-art language-guided methods when limited text is available.