FGAseg: Fine-Grained Pixel-Text Alignment for Open-Vocabulary Semantic Segmentation

TL;DR

FGAseg introduces a novel approach for open-vocabulary semantic segmentation by enhancing pixel-text alignment and boundary information using cross-modal attention and pseudo-masks, significantly improving performance over existing methods.

Contribution

The paper presents FGAseg, a model that achieves fine-grained pixel-text alignment and boundary supplementation, addressing limitations of VLMs in segmentation tasks.

Findings

Outperforms existing methods on open-vocabulary segmentation benchmarks

Effectively refines coarse CLIP alignment to pixel-level detail

Enhances boundary detection with pseudo-masks derived from similarity measures

Abstract

Open-vocabulary segmentation aims to identify and segment specific regions and objects based on text-based descriptions. A common solution is to leverage powerful vision-language models (VLMs), such as CLIP, to bridge the gap between vision and text information. However, VLMs are typically pretrained for image-level vision-text alignment, focusing on global semantic features. In contrast, segmentation tasks require fine-grained pixel-level alignment and detailed category boundary information, which VLMs alone cannot provide. As a result, information extracted directly from VLMs can't meet the requirements of segmentation tasks. To address this limitation, we propose FGAseg, a model designed for fine-grained pixel-text alignment and category boundary supplementation. The core of FGAseg is a Pixel-Level Alignment module that employs a cross-modal attention mechanism and a text-pixel alignment loss to refine the coarse-grained alignment from CLIP, achieving finer-grained pixel-text semantic alignment. Additionally, to enrich category boundary information, we introduce the alignment matrices as optimizable pseudo-masks during forward propagation and propose Category Information Supplementation module. These pseudo-masks, derived from cosine and convolutional similarity, provide essential global and local boundary information between different categories. By combining these two strategies, FGAseg effectively enhances pixel-level alignment and category boundary information, addressing key challenges in open-vocabulary segmentation. Extensive experiments demonstrate that FGAseg outperforms existing methods on open-vocabulary semantic segmentation benchmarks.