DreamMask: Boosting Open-vocabulary Panoptic Segmentation with Synthetic Data

TL;DR

DreamMask introduces a data-centric approach for open-vocabulary panoptic segmentation by generating synthetic training data and aligning it with real data, significantly improving model generalization and performance.

Contribution

It presents a systematic data generation pipeline and a synthetic-real alignment loss, enhancing open-vocabulary segmentation without extensive manual data collection.

Findings

Outperforms previous state-of-the-art by 2.1% mIoU on ADE20K.

Synthetic data can surpass manually collected web data.

The approach simplifies large-scale data collection for segmentation tasks.

Abstract

Open-vocabulary panoptic segmentation has received significant attention due to its applicability in the real world. Despite claims of robust generalization, we find that the advancements of previous works are attributed mainly on trained categories, exposing a lack of generalization to novel classes. In this paper, we explore boosting existing models from a data-centric perspective. We propose DreamMask, which systematically explores how to generate training data in the open-vocabulary setting, and how to train the model with both real and synthetic data. For the first part, we propose an automatic data generation pipeline with off-the-shelf models. We propose crucial designs for vocabulary expansion, layout arrangement, data filtering, etc. Equipped with these techniques, our generated data could significantly outperform the manually collected web data. To train the model with generated data, a synthetic-real alignment loss is designed to bridge the representation gap, bringing noticeable improvements across multiple benchmarks. In general, DreamMask significantly simplifies the collection of large-scale training data, serving as a plug-and-play enhancement for existing methods. For instance, when trained on COCO and tested on ADE20K, the model equipped with DreamMask outperforms the previous state-of-the-art by a substantial margin of 2.1% mIoU.