Prompting classes: Exploring the Power of Prompt Class Learning in Weakly Supervised Semantic Segmentation

TL;DR

This paper investigates the use of prompt class learning in weakly supervised semantic segmentation, revealing that simple prompt modifications can significantly improve segmentation performance and achieve state-of-the-art results.

Contribution

It introduces the POLE strategy, a novel prompt learning method that enhances weakly supervised semantic segmentation by optimizing class tokens, demonstrating superior performance.

Findings

Modifying only the class token greatly impacts CAM.

The class token linked to ground truth may not produce the best CAM.

POLE achieves state-of-the-art results on WSSS benchmarks.

Abstract

Recently, CLIP-based approaches have exhibited remarkable performance on generalization and few-shot learning tasks, fueled by the power of contrastive language-vision pre-training. In particular, prompt tuning has emerged as an effective strategy to adapt the pre-trained language-vision models to downstream tasks by employing task-related textual tokens. Motivated by this progress, in this work we question whether other fundamental problems, such as weakly supervised semantic segmentation (WSSS), can benefit from prompt tuning. Our findings reveal two interesting observations that shed light on the impact of prompt tuning on WSSS. First, modifying only the class token of the text prompt results in a greater impact on the Class Activation Map (CAM), compared to arguably more complex strategies that optimize the context. And second, the class token associated with the image ground truth does not necessarily correspond to the category that yields the best CAM. Motivated by these observations, we introduce a novel approach based on a PrOmpt cLass lEarning (POLE) strategy. Through extensive experiments we demonstrate that our simple, yet efficient approach achieves SOTA performance in a well-known WSSS benchmark. These results highlight not only the benefits of language-vision models in WSSS but also the potential of prompt learning for this problem. The code is available at https://github.com/rB080/WSS_POLE.