LoGoSeg: Integrating Local and Global Features for Open-Vocabulary Semantic Segmentation

TL;DR

LoGoSeg introduces a novel framework for open-vocabulary semantic segmentation that combines local and global features with object priors and region-aware alignment, achieving efficient and accurate pixel-wise segmentation for seen and unseen categories.

Contribution

It presents a single-stage method integrating object priors, region-level alignment, and dual-stream fusion, eliminating the need for external proposals or additional datasets.

Findings

Competitive performance on six benchmarks

Strong generalization to unseen categories

Efficient single-stage framework

Abstract

Open-vocabulary semantic segmentation (OVSS) extends traditional closed-set segmentation by enabling pixel-wise annotation for both seen and unseen categories using arbitrary textual descriptions. While existing methods leverage vision-language models (VLMs) like CLIP, their reliance on image-level pretraining often results in imprecise spatial alignment, leading to mismatched segmentations in ambiguous or cluttered scenes. However, most existing approaches lack strong object priors and region-level constraints, which can lead to object hallucination or missed detections, further degrading performance. To address these challenges, we propose LoGoSeg, an efficient single-stage framework that integrates three key innovations: (i) an object existence prior that dynamically weights relevant categories through global image-text similarity, effectively reducing hallucinations; (ii) a region-aware alignment module that establishes precise region-level visual-textual correspondences; and (iii) a dual-stream fusion mechanism that optimally combines local structural information with global semantic context. Unlike prior works, LoGoSeg eliminates the need for external mask proposals, additional backbones, or extra datasets, ensuring efficiency. Extensive experiments on six benchmarks (A-847, PC-459, A-150, PC-59, PAS-20, and PAS-20b) demonstrate its competitive performance and strong generalization in open-vocabulary settings.