MTA-CLIP: Language-Guided Semantic Segmentation with Mask-Text Alignment

TL;DR

MTA-CLIP introduces a mask-level vision-language alignment framework that enhances semantic segmentation by improving class boundary clarity and detailed pixel-level correspondence, surpassing previous methods on benchmark datasets.

Contribution

The paper presents a novel mask-text decoder and contrastive learning approach for improved alignment in vision-language models for segmentation.

Findings

Achieves state-of-the-art performance on ADE20k and Cityscapes datasets.

Surpasses prior methods by 2.8% and 1.3% on benchmark datasets.

Introduces mask-text contrastive learning and prompt learning for better class representation.

Abstract

Recent approaches have shown that large-scale vision-language models such as CLIP can improve semantic segmentation performance. These methods typically aim for pixel-level vision-language alignment, but often rely on low resolution image features from CLIP, resulting in class ambiguities along boundaries. Moreover, the global scene representations in CLIP text embeddings do not directly correlate with the local and detailed pixel-level features, making meaningful alignment more difficult. To address these limitations, we introduce MTA-CLIP, a novel framework employing mask-level vision-language alignment. Specifically, we first propose Mask-Text Decoder that enhances the mask representations using rich textual data with the CLIP language model. Subsequently, it aligns mask representations with text embeddings using Mask-to-Text Contrastive Learning. Furthermore, we introduce MaskText Prompt Learning, utilizing multiple context-specific prompts for text embeddings to capture diverse class representations across masks. Overall, MTA-CLIP achieves state-of-the-art, surpassing prior works by an average of 2.8% and 1.3% on on standard benchmark datasets, ADE20k and Cityscapes, respectively.