InvSeg: Test-Time Prompt Inversion for Semantic Segmentation

TL;DR

InvSeg introduces a test-time prompt inversion technique that enhances open-vocabulary semantic segmentation by aligning visual and textual features through structure-aware prompt enrichment, achieving state-of-the-art results.

Contribution

The paper proposes InvSeg, a novel method that inverts image-specific visual context into text prompts, improving semantic segmentation accuracy across diverse datasets.

Findings

Achieves state-of-the-art performance on PASCAL VOC, PASCAL Context, and COCO datasets.

Utilizes Contrastive Soft Clustering to improve mask distinction and internal consistency.

Effectively aligns visual and textual features for open-vocabulary segmentation.

Abstract

Visual-textual correlations in the attention maps derived from text-to-image diffusion models are proven beneficial to dense visual prediction tasks, e.g., semantic segmentation. However, a significant challenge arises due to the input distributional discrepancy between the context-rich sentences used for image generation and the isolated class names typically used in semantic segmentation. This discrepancy hinders diffusion models from capturing accurate visual-textual correlations. To solve this, we propose InvSeg, a test-time prompt inversion method that tackles open-vocabulary semantic segmentation by inverting image-specific visual context into text prompt embedding space, leveraging structure information derived from the diffusion model's reconstruction process to enrich text prompts so as to associate each class with a structure-consistent mask. Specifically, we introduce Contrastive Soft Clustering (CSC) to align derived masks with the image's structure information, softly selecting anchors for each class and calculating weighted distances to push inner-class pixels closer while separating inter-class pixels, thereby ensuring mask distinction and internal consistency. By incorporating sample-specific context, InvSeg learns context-rich text prompts in embedding space and achieves accurate semantic alignment across modalities. Experiments show that InvSeg achieves state-of-the-art performance on the PASCAL VOC, PASCAL Context and COCO Object datasets.