Show or Tell? A Benchmark To Evaluate Visual and Textual Prompts in Semantic Segmentation

TL;DR

This paper introduces Show or Tell (SoT), a benchmark for comparing visual and textual prompts in semantic segmentation across diverse datasets, revealing their respective strengths and weaknesses.

Contribution

The paper presents the first comprehensive benchmark evaluating both visual and textual prompts in semantic segmentation under identical conditions.

Findings

Open-vocabulary methods perform well on common, easily described concepts.

Visual reference prompts show high variability depending on input quality.

Complex domains like tools challenge open-vocabulary approaches.

Abstract

Prompt engineering has shown remarkable success with large language models, yet its systematic exploration in computer vision remains limited. In semantic segmentation, both textual and visual prompts offer distinct advantages: textual prompts through open-vocabulary methods allow segmentation of arbitrary categories, while visual reference prompts provide intuitive reference examples. However, existing benchmarks evaluate these modalities in isolation, without direct comparison under identical conditions. We present Show or Tell (SoT), a novel benchmark specifically designed to evaluate both visual and textual prompts for semantic segmentation across 14 datasets spanning 7 diverse domains (common scenes, urban, food, waste, parts, tools, and land-cover). We evaluate 5 open-vocabulary methods and 4 visual reference prompt approaches, adapting the latter to handle multi-class segmentation through a confidence-based mask merging strategy. Our extensive experiments reveal that open-vocabulary methods excel with common concepts easily described by text but struggle with complex domains like tools, while visual reference prompt methods achieve good average results but exhibit high variability depending on the input prompt. Through comprehensive quantitative and qualitative analysis, we identify the strengths and weaknesses of both prompting modalities, providing valuable insights to guide future research in vision foundation models for segmentation tasks.