Mitigating Objectness Bias and Region-to-Text Misalignment for Open-Vocabulary Panoptic Segmentation

TL;DR

This paper introduces OVRCOAT, a modular framework that improves open-vocabulary panoptic segmentation by addressing objectness bias and regional misalignment, leading to state-of-the-art results on multiple datasets.

Contribution

The paper proposes a novel CLIP-conditioned objectness adjustment and mask-to-text refinement to enhance segmentation of unseen categories without extensive fine-tuning.

Findings

Achieved +5.5% PQ on ADE20K

Improved PQ by +7.1% on Mapillary Vistas

Enhanced PQ by +3% on Cityscapes

Abstract

Open-vocabulary panoptic segmentation remains hindered by two coupled issues: (i) mask selection bias, where objectness heads trained on closed vocabularies suppress masks of categories not observed in training, and (ii) limited regional understanding in vision-language models such as CLIP, which were optimized for global image classification rather than localized segmentation. We introduce OVRCOAT, a simple, modular framework that tackles both. First, a CLIP-conditioned objectness adjustment (COAT) updates background/foreground probabilities, preserving high-quality masks for out-of-vocabulary objects. Second, an open-vocabulary mask-to-text refinement (OVR) strengthens CLIP's region-level alignment to improve classification of both seen and unseen classes with markedly lower memory cost than prior fine-tuning schemes. The two components combine to jointly improve objectness estimation and mask recognition, yielding consistent panoptic gains. Despite its simplicity, OVRCOAT sets a new state of the art on ADE20K (+5.5% PQ) and delivers clear gains on Mapillary Vistas and Cityscapes (+7.1% and +3% PQ, respectively). The code is available at: https://github.com/nickormushev/OVRCOAT