VAQUUM: Are Vague Quantifiers Grounded in Visual Data?

TL;DR

This paper introduces VAQUUM, a dataset of human ratings on vague quantifiers in images, and evaluates how well vision-and-language models align with human judgments, revealing both similarities and inconsistencies.

Contribution

The work provides a new dataset, VAQUUM, and systematically compares VLMs to humans in understanding vague quantifiers in visual contexts, highlighting differences in judgment and production processes.

Findings

VLMs are influenced by object counts similar to humans.

Significant inconsistencies exist across models in different evaluation settings.

Judging and producing vague quantifiers involve different cognitive processes.

Abstract

Vague quantifiers such as "a few" and "many" are influenced by various contextual factors, including the number of objects present in a given context. In this work, we evaluate the extent to which vision-and-language models (VLMs) are compatible with humans when producing or judging the appropriateness of vague quantifiers in visual contexts. We release a novel dataset, VAQUUM, containing 20,300 human ratings on quantified statements across a total of 1089 images. Using this dataset, we compare human judgments and VLM predictions using three different evaluation methods. Our findings show that VLMs, like humans, are influenced by object counts in vague quantifier use. However, we find significant inconsistencies across models in different evaluation settings, suggesting that judging and producing vague quantifiers rely on two different processes.