SpaceVLM: Sub-Space Modeling of Negation in Vision-Language Models

TL;DR

This paper introduces a training-free subspace modeling approach in vision-language models to improve negation understanding, significantly enhancing performance without sacrificing zero-shot capabilities.

Contribution

It proposes a novel subspace-based framework for negation in VLMs that does not require retraining, maintaining zero-shot performance while improving negation comprehension.

Findings

Improves negation understanding by about 30% on average.

Closes the gap between affirmative and negated prompts.

Preserves zero-shot performance of models.

Abstract

Vision-Language Models (VLMs) struggle with negation. Given a prompt like "retrieve (or generate) a street scene without pedestrians," they often fail to respect the "not." Existing methods address this limitation by fine-tuning on large negation datasets, but such retraining often compromises the model's zero-shot performance on affirmative prompts. We show that the embedding space of VLMs, such as CLIP, can be divided into semantically consistent subspaces. Based on this property, we propose a training-free framework that models negation as a subspace in the joint embedding space rather than a single point (Figure 1). To find the matching image for a caption such as "A but not N," we construct two spherical caps around the embeddings of A and N, and we score images by the central direction of the region that is close to A and far from N. Across retrieval, MCQ, and text-to-image tasks, our method improves negation understanding by about 30% on average over prior methods. It closes the gap between affirmative and negated prompts while preserving the zero-shot performance that fine-tuned models fail to maintain. Code will be released upon publication.