Prompt Sensitivity in Vision-Language Grounding: How Small Changes in Wording Affect Object Detection

TL;DR

This paper investigates how small wording changes in prompts affect object detection in vision-language models, revealing significant instability and structured variability in object grounding results.

Contribution

It demonstrates that prompt wording significantly impacts object selection, with instability driven by the argmax mechanism rather than text similarity alone.

Findings

Overlapping prompts select different objects with mean instability of 2.11 instances.

Prompt ensembling does not improve grounding quality and shifts selections.

Text embedding proximity explains only 34% of grounding disagreement.

Abstract

Vision-language models enable open-vocabulary object grounding through natural language queries, under the implicit assumption that semantically equivalent descriptions yield consistent outputs. We examine this assumption using a controlled pipeline combining DETR for object proposals with CLIP for language-conditioned selection on 263 COCO val2017 images. We find that overlapping prompts such as "a person," "a human," and "a pedestrian" frequently select different instances, with mean instability of 2.11 distinct selections across six prompts. PCA analysis shows this variability is structured and directional, not random. Prompt ensembling does not improve quality and often shifts selections toward generic regions. We further show that text embedding proximity explains only 34% of grounding disagreement (r = -0.58), confirming that instability arises from the argmax selection mechanism rather than text-level distances alone.