Distilling Spectral Graph for Object-Context Aware Open-Vocabulary Semantic Segmentation

TL;DR

This paper introduces a spectral graph distillation method that incorporates object-level context into vision-language models for open-vocabulary semantic segmentation, improving intra-object consistency and alignment with arbitrary queries.

Contribution

The work presents a novel spectral distillation approach that enhances object-level contextual understanding in vision models for open-vocabulary segmentation tasks.

Findings

Achieves state-of-the-art performance on multiple datasets.

Improves intra-object semantic coherence.

Enhances alignment of text embeddings with object presence.

Abstract

Open-Vocabulary Semantic Segmentation (OVSS) has advanced with recent vision-language models (VLMs), enabling segmentation beyond predefined categories through various learning schemes. Notably, training-free methods offer scalable, easily deployable solutions for handling unseen data, a key goal of OVSS. Yet, a critical issue persists: lack of object-level context consideration when segmenting complex objects in the challenging environment of OVSS based on arbitrary query prompts. This oversight limits models' ability to group semantically consistent elements within object and map them precisely to user-defined arbitrary classes. In this work, we introduce a novel approach that overcomes this limitation by incorporating object-level contextual knowledge within images. Specifically, our model enhances intra-object consistency by distilling spectral-driven features from vision foundation models into the attention mechanism of the visual encoder, enabling semantically coherent components to form a single object mask. Additionally, we refine the text embeddings with zero-shot object presence likelihood to ensure accurate alignment with the specific objects represented in the images. By leveraging object-level contextual knowledge, our proposed approach achieves state-of-the-art performance with strong generalizability across diverse datasets.