DEARLi: Decoupled Enhancement of Recognition and Localization for Semi-supervised Panoptic Segmentation

TL;DR

DEARLi introduces a novel semi-supervised panoptic segmentation method that decouples recognition and localization, leveraging foundation models to significantly improve performance with limited labeled data and reduced computational resources.

Contribution

The paper proposes DEARLi, a new approach that enhances recognition and localization separately using foundation models, achieving state-of-the-art results in semi-supervised segmentation with less memory.

Findings

Outperforms state-of-the-art in semi-supervised semantic segmentation.

Achieves 29.9 PQ and 38.9 mIoU on ADE20K with only 158 labeled images.

Requires 8x less GPU memory than previous methods.

Abstract

Pixel-level annotation is expensive and time-consuming. Semi-supervised segmentation methods address this challenge by learning models on few labeled images alongside a large corpus of unlabeled images. Although foundation models could further account for label scarcity, effective mechanisms for their exploitation remain underexplored. We address this by devising a novel semi-supervised panoptic approach fueled by two dedicated foundation models. We enhance recognition by complementing unsupervised mask-transformer consistency with zero-shot classification of CLIP features. We enhance localization by class-agnostic decoder warm-up with respect to SAM pseudo-labels. The resulting decoupled enhancement of recognition and localization (DEARLi) particularly excels in the most challenging semi-supervised scenarios with large taxonomies and limited labeled data. Moreover, DEARLi outperforms the state of the art in semi-supervised semantic segmentation by a large margin while requiring 8x less GPU memory, in spite of being trained only for the panoptic objective. We observe 29.9 PQ and 38.9 mIoU on ADE20K with only 158 labeled images. The source code is available at https://github.com/helen1c/DEARLi.