TAMISeg: Text-Aligned Multi-scale Medical Image Segmentation with Semantic Encoder Distillation

TL;DR

TAMISeg is a novel text-guided medical image segmentation framework that leverages semantic distillation and multi-scale decoding to improve segmentation accuracy with limited annotations.

Contribution

It introduces a multi-component framework combining a robust encoder, semantic distillation, and scale-adaptive decoding for enhanced medical image segmentation.

Findings

Outperforms existing methods on multiple datasets.

Demonstrates robustness to image noise and low contrast.

Achieves superior segmentation accuracy with limited annotations.

Abstract

Medical image segmentation remains challenging due to limited fine-grained annotations, complex anatomical structures, and image degradation from noise, low contrast, or illumination variation. We propose TAMISeg, a text-guided segmentation framework that incorporates clinical language prompts and semantic distillation as auxiliary semantic cues to enhance visual understanding and reduce reliance on pixel-level fine-grained annotations. TAMISeg integrates three core components: a consistency-aware encoder pretrained with strong perturbations for robust feature extraction, a semantic encoder distillation module with supervision from a frozen DINOv3 teacher to enhance semantic discriminability, and a scale-adaptive decoder that segments anatomical structures across different spatial scales. Experiments on the Kvasir-SEG, MosMedData+, and QaTa-COV19 datasets demonstrate that TAMISeg consistently outperforms existing uni-modal and multi-modal methods in both qualitative and quantitative evaluations. Code will be made publicly available at https://github.com/qczggaoqiang/TAMISeg.