Test-Time Adaptation of Vision-Language Models for Open-Vocabulary Semantic Segmentation

TL;DR

This paper introduces a novel test-time adaptation method for open-vocabulary semantic segmentation using vision-language models, improving performance without additional training data across diverse datasets and conditions.

Contribution

It proposes a multi-level, multi-prompt entropy minimization approach tailored for segmentation, and establishes a comprehensive benchmark suite for evaluating TTA in open-vocabulary segmentation.

Findings

Our method outperforms TTA classification baselines across multiple datasets.

It remains effective with a single test sample and no extra training.

The benchmark suite includes 87 diverse test scenarios.

Abstract

Recently, test-time adaptation has attracted wide interest in the context of vision-language models for image classification. However, to the best of our knowledge, the problem is completely overlooked in dense prediction tasks such as Open-Vocabulary Semantic Segmentation (OVSS). In response, we propose a novel TTA method tailored to adapting VLMs for segmentation during test time. Unlike TTA methods for image classification, our Multi-Level and Multi-Prompt (MLMP) entropy minimization integrates features from intermediate vision-encoder layers and is performed with different text-prompt templates at both the global CLS token and local pixel-wise levels. Our approach could be used as plug-and-play for any segmentation network, does not require additional training data or labels, and remains effective even with a single test sample. Furthermore, we introduce a comprehensive OVSS TTA benchmark suite, which integrates a rigorous evaluation protocol, nine segmentation datasets, 15 common synthetic corruptions, and additional real and rendered domain shifts, \textbf{with a total of 87 distinct test scenarios}, establishing a standardized and comprehensive testbed for future TTA research in open-vocabulary segmentation. Our experiments on this suite demonstrate that our segmentation-tailored method consistently delivers significant gains over direct adoption of TTA classification baselines. Code and data are available at https://github.com/dosowiechi/MLMP.