Class Enhancement Losses with Pseudo Labels for Zero-shot Semantic Segmentation

TL;DR

This paper introduces class enhancement losses and a pseudo label generation pipeline leveraging semantic relationships and vision-language models to improve zero-shot semantic segmentation performance.

Contribution

It proposes novel class enhancement losses and a pseudo label pipeline that bypass background embedding and exploit semantic relationships for better zero-shot segmentation.

Findings

Achieves state-of-the-art results on benchmark datasets.

Effectively captures relationships between seen and unseen classes.

Flexible approach applicable to open-vocabulary segmentation.

Abstract

Recent mask proposal models have significantly improved the performance of zero-shot semantic segmentation. However, the use of a `background' embedding during training in these methods is problematic as the resulting model tends to over-learn and assign all unseen classes as the background class instead of their correct labels. Furthermore, they ignore the semantic relationship of text embeddings, which arguably can be highly informative for zero-shot prediction as seen classes may have close relationship with unseen classes. To this end, this paper proposes novel class enhancement losses to bypass the use of the background embbedding during training, and simultaneously exploit the semantic relationship between text embeddings and mask proposals by ranking the similarity scores. To further capture the relationship between seen and unseen classes, we propose an effective pseudo label generation pipeline using pretrained vision-language model. Extensive experiments on several benchmark datasets show that our method achieves overall the best performance for zero-shot semantic segmentation. Our method is flexible, and can also be applied to the challenging open-vocabulary semantic segmentation problem.