From Synthetic Scenes to Real Performance: Enhancing Spatial Reasoning in VLMs

TL;DR

This paper introduces a bias-controlled synthetic data generation method for fine-tuning vision-language models, significantly improving their spatial reasoning and transferability to real-world data.

Contribution

It proposes a novel synthetic data creation process that eliminates bias and distribution issues, enhancing VLM fine-tuning and real-world performance.

Findings

Balanced synthetic data reduces bias and overfitting.

Fine-tuning on synthetic data improves real-world accuracy by 13%.

Synthetic data outperforms full real-world dataset in spatial reasoning tasks.

Abstract

Fine-tuning Vision-Language Models (VLMs) is a common strategy to improve performance following an ad-hoc data collection and annotation of real-world scenes. However, this process is often prone to biases, errors, and distribution imbalance, resulting in overfitting and imbalanced performance. Although a few studies have tried to address this problem by generating synthetic data, they lacked control over distribution bias and annotation quality. To address these challenges, we redesign the fine-tuning process in two ways. First, we control the generation of data and its annotations, ensuring it is free from bias, distribution imbalance, and annotation errors. We automatically construct the dataset by comprehensively sampling objects' attributes, including color, shape, size, and position within the scene. Secondly, using this annotated dataset, we fine-tune state-of-the-art VLMs and assess performance transferability to real-world data on the absolute position task. We conduct exhaustive evaluations on both synthetic and real-world benchmarks. Our experiments reveal two key findings: 1) fine-tuning on balanced synthetic data yields uniform performance across the visual scene and mitigates common biases; and 2) fine-tuning on synthetic stimuli improves performance by 13% on real-world data (COCO), outperforming models fine-tuned on the full COCO train set.