Test-Time Optimization for Domain Adaptive Open Vocabulary Segmentation

TL;DR

Seg-TTO introduces a test-time optimization framework for open-vocabulary semantic segmentation that significantly improves performance on domain-specific datasets by aligning model parameters with input images using a novel self-supervised objective.

Contribution

It proposes a novel test-time optimization method, Seg-TTO, that enhances zero-shot open-vocabulary segmentation in specialized domains, outperforming existing approaches.

Findings

Up to 27% mIoU improvement on challenging datasets

Effective integration with state-of-the-art OVSS methods

Establishes new state-of-the-art performance in domain-specific OVSS tasks

Abstract

We present Seg-TTO, a novel framework for zero-shot, open-vocabulary semantic segmentation (OVSS), designed to excel in specialized domain tasks. While current open-vocabulary approaches show impressive performance on standard segmentation benchmarks under zero-shot settings, they fall short of supervised counterparts on highly domain-specific datasets. We focus on segmentation-specific test-time optimization to address this gap. Segmentation requires an understanding of multiple concepts within a single image while retaining the locality and spatial structure of representations. We propose a novel self-supervised objective adhering to these requirements and use it to align the model parameters with input images at test time. In the textual modality, we learn multiple embeddings for each category to capture diverse concepts within an image, while in the visual modality, we calculate pixel-level losses followed by embedding aggregation operations specific to preserving spatial structure. Our resulting framework termed Seg-TTO is a plug-and-play module. We integrate Seg-TTO with three state-of-the-art OVSS approaches and evaluate across 22 challenging OVSS tasks covering a range of specialized domains. Our Seg-TTO demonstrates clear performance improvements (up to 27% mIoU increase on some datasets) establishing new state-of-the-art. Our code and models will be released publicly.