SITUATE -- Synthetic Object Counting Dataset for VLM training

TL;DR

SITUATE is a new dataset for training and evaluating vision-language models on counting tasks with spatial constraints, improving generalization to out-of-distribution images.

Contribution

The paper introduces SITUATE, a novel dataset that enhances counting model training by providing controlled spatial and occlusion information.

Findings

Finetuning Qwen VL 2.5 7B on SITUATE improves accuracy on Pixmo count test data.

SITUATE helps models generalize better to out-of-distribution images.

The dataset bridges the gap between simple 2D datasets and ambiguous real-world datasets.

Abstract

We present SITUATE, a novel dataset designed for training and evaluating Vision Language Models on counting tasks with spatial constraints. The dataset bridges the gap between simple 2D datasets like VLMCountBench and often ambiguous real-life datasets like TallyQA, which lack control over occlusions and spatial composition. Experiments show that our dataset helps to improve generalization for out-of-distribution images, since a finetune of Qwen VL 2.5 7B on SITUATE improves accuracy on the Pixmo count test data, but not vice versa. We cross validate this by comparing the model performance across established other counting benchmarks and against an equally sized fine-tuning set derived from Pixmo count.