Open-Vocabulary Semantic Segmentation with Image Embedding Balancing

TL;DR

This paper introduces EBSeg, a novel framework for open-vocabulary semantic segmentation that balances image embeddings and enforces semantic structure consistency, significantly improving generalization to new classes.

Contribution

The paper proposes EBSeg with an Adaptively Balanced Decoder and SSC Loss, enhancing CLIP-based segmentation by balancing embeddings and aligning semantic structures for better generalization.

Findings

Outperforms state-of-the-art methods on various benchmarks

Effectively balances training and new class recognition

Improves semantic structure understanding in segmentation tasks

Abstract

Open-vocabulary semantic segmentation is a challenging task, which requires the model to output semantic masks of an image beyond a close-set vocabulary. Although many efforts have been made to utilize powerful CLIP models to accomplish this task, they are still easily overfitting to training classes due to the natural gaps in semantic information between training and new classes. To overcome this challenge, we propose a novel framework for openvocabulary semantic segmentation called EBSeg, incorporating an Adaptively Balanced Decoder (AdaB Decoder) and a Semantic Structure Consistency loss (SSC Loss). The AdaB Decoder is designed to generate different image embeddings for both training and new classes. Subsequently, these two types of embeddings are adaptively balanced to fully exploit their ability to recognize training classes and generalization ability for new classes. To learn a consistent semantic structure from CLIP, the SSC Loss aligns the inter-classes affinity in the image feature space with that in the text feature space of CLIP, thereby improving the generalization ability of our model. Furthermore, we employ a frozen SAM image encoder to complement the spatial information that CLIP features lack due to the low training image resolution and image-level supervision inherent in CLIP. Extensive experiments conducted across various benchmarks demonstrate that the proposed EBSeg outperforms the state-of-the-art methods. Our code and trained models will be here: https://github.com/slonetime/EBSeg.