Expand VSR Benchmark for VLLM to Expertize in Spatial Rules

TL;DR

This paper enhances the evaluation and training of Vision Large Language Models (VLLMs) for visual spatial reasoning by expanding datasets, integrating multiple visual encoders, and using diffusion models to improve positional understanding, resulting in a significantly more accurate model.

Contribution

The authors expanded the VSR benchmark with controllably generated spatial data and integrated multiple visual encoders, creating a VLLM that excels in visual positional reasoning.

Findings

VLLMs show over-sensitivity to language and under-sensitivity to visual position.

Expanded datasets and model structures improve positional reasoning accuracy.

VSRE achieved over 27% higher accuracy on the VSR test set.

Abstract

Distinguishing spatial relations is a basic part of human cognition which requires fine-grained perception on cross-instance. Although benchmarks like MME, MMBench and SEED comprehensively have evaluated various capabilities which already include visual spatial reasoning(VSR). There is still a lack of sufficient quantity and quality evaluation and optimization datasets for Vision Large Language Models(VLLMs) specifically targeting visual positional reasoning. To handle this, we first diagnosed current VLLMs with the VSR dataset and proposed a unified test set. We found current VLLMs to exhibit a contradiction of over-sensitivity to language instructions and under-sensitivity to visual positional information. By expanding the original benchmark from two aspects of tunning data and model structure, we mitigated this phenomenon. To our knowledge, we expanded spatially positioned image data controllably using diffusion models for the first time and integrated original visual encoding(CLIP) with other 3 powerful visual encoders(SigLIP, SAM and DINO). After conducting combination experiments on scaling data and models, we obtained a VLLM VSR Expert(VSRE) that not only generalizes better to different instructions but also accurately distinguishes differences in visual positional information. VSRE achieved over a 27\% increase in accuracy on the VSR test set. It becomes a performant VLLM on the position reasoning of both the VSR dataset and relevant subsets of other evaluation benchmarks. We open-sourced the expanded model with data and Appendix at \url{https://github.com/peijin360/vsre} and hope it will accelerate advancements in VLLM on VSR learning.