Open Vocabulary Panoptic Segmentation With Retrieval Augmentation

TL;DR

This paper introduces RetCLIP, a retrieval-augmented method for open vocabulary panoptic segmentation that enhances the ability to segment unseen classes by combining feature retrieval with CLIP-based classification, significantly improving performance.

Contribution

The paper proposes RetCLIP, a novel retrieval-augmented approach that improves open vocabulary panoptic segmentation, especially for unseen classes, by leveraging a masked segment feature database and combining retrieval with CLIP scores.

Findings

Achieves 30.9 PQ on ADE20k, outperforming baseline by 4.5 points.

Demonstrates improved mAP and mIoU metrics, indicating better segmentation quality.

Effectively generalizes to unseen classes beyond training data.

Abstract

Given an input image and set of class names, panoptic segmentation aims to label each pixel in an image with class labels and instance labels. In comparison, Open Vocabulary Panoptic Segmentation aims to facilitate the segmentation of arbitrary classes according to user input. The challenge is that a panoptic segmentation system trained on a particular dataset typically does not generalize well to unseen classes beyond the training data. In this work, we propose RetCLIP, a retrieval-augmented panoptic segmentation method that improves the performance of unseen classes. In particular, we construct a masked segment feature database using paired image-text data. At inference time, we use masked segment features from the input image as query keys to retrieve similar features and associated class labels from the database. Classification scores for the masked segment are assigned based on the similarity between query features and retrieved features. The retrieval-based classification scores are combined with CLIP-based scores to produce the final output. We incorporate our solution with a previous SOTA method (FC-CLIP). When trained on COCO, the proposed method demonstrates 30.9 PQ, 19.3 mAP, 44.0 mIoU on the ADE20k dataset, achieving +4.5 PQ, +2.5 mAP, +10.0 mIoU absolute improvement over the baseline.