Split Matching for Inductive Zero-shot Semantic Segmentation

TL;DR

This paper introduces Split Matching, a novel assignment strategy for zero-shot semantic segmentation that decouples matching for seen and unseen classes, improving performance without relying on full supervision.

Contribution

The paper proposes Split Matching, a new decoupled Hungarian matching method for inductive ZSS, and a Multi-scale Feature Enhancement module to improve spatial detail capture.

Findings

Achieves state-of-the-art results on standard benchmarks.

Effectively distinguishes unseen categories without full supervision.

Enhances spatial detail understanding through multi-scale features.

Abstract

Zero-shot Semantic Segmentation (ZSS) aims to segment categories that are not annotated during training. While fine-tuning vision-language models has achieved promising results, these models often overfit to seen categories due to the lack of supervision for unseen classes. As an alternative to fully supervised approaches, query-based segmentation has shown great latent in ZSS, as it enables object localization without relying on explicit labels. However, conventional Hungarian matching, a core component in query-based frameworks, needs full supervision and often misclassifies unseen categories as background in the setting of ZSS. To address this issue, we propose Split Matching (SM), a novel assignment strategy that decouples Hungarian matching into two components: one for seen classes in annotated regions and another for latent classes in unannotated regions (referred to as unseen candidates). Specifically, we partition the queries into seen and candidate groups, enabling each to be optimized independently according to its available supervision. To discover unseen candidates, we cluster CLIP dense features to generate pseudo masks and extract region-level embeddings using CLS tokens. Matching is then conducted separately for the two groups based on both class-level similarity and mask-level consistency. Additionally, we introduce a Multi-scale Feature Enhancement (MFE) module that refines decoder features through residual multi-scale aggregation, improving the model's ability to capture spatial details across resolutions. SM is the first to introduce decoupled Hungarian matching under the inductive ZSS setting, and achieves state-of-the-art performance on two standard benchmarks.