Spatial-aware Symmetric Alignment for Text-guided Medical Image Segmentation

TL;DR

This paper introduces the Spatial-aware Symmetric Alignment framework that improves text-guided medical image segmentation by capturing spatial and descriptive information, leading to more accurate lesion localization.

Contribution

It proposes a novel symmetric optimal transport alignment and a spatial guidance strategy to better associate text and image regions, especially with spatial constraints.

Findings

Achieves state-of-the-art segmentation accuracy on benchmarks.

Effectively captures spatial relations in lesion segmentation.

Improves handling of complex descriptive and locational texts.

Abstract

Text-guided Medical Image Segmentation has shown considerable promise for medical image segmentation, with rich clinical text serving as an effective supplement for scarce data. However, current methods have two key bottlenecks. On one hand, they struggle to process diagnostic and descriptive texts simultaneously, making it difficult to identify lesions and establish associations with image regions. On the other hand, existing approaches focus on lesions description and fail to capture positional constraints, leading to critical deviations. Specifically, with the text "in the left lower lung", the segmentation results may incorrectly cover both sides of the lung. To address the limitations, we propose the Spatial-aware Symmetric Alignment (SSA) framework to enhance the capacity of referring hybrid medical texts consisting of locational, descriptive, and diagnostic information. Specifically, we propose symmetric optimal transport alignment mechanism to strengthen the associations between image regions and multiple relevant expressions, which establishes bi-directional fine-grained multimodal correspondences. In addition, we devise a composite directional guidance strategy that explicitly introduces spatial constraints in the text by constructing region-level guidance masks. Extensive experiments on public benchmarks demonstrate that SSA achieves state-of-the-art (SOTA) performance, particularly in accurately segmenting lesions characterized by spatial relational constraints.