Scale Can't Overcome Pragmatics: The Impact of Reporting Bias on Vision-Language Reasoning

TL;DR

This paper investigates how reporting bias in training data limits the reasoning abilities of vision-language models, showing that increasing data or model size does not improve reasoning skills without targeted data curation.

Contribution

It reveals that reporting bias suppresses key reasoning skills in VLMs and demonstrates that targeted annotation improves reasoning performance, challenging the reliance on scale alone.

Findings

VLMs perform poorly on spatial, temporal, negation, and counting reasoning tasks.

Scaling data and models does not automatically improve reasoning skills.

Targeted annotations effectively enhance reasoning capabilities.

Abstract

The lack of reasoning capabilities in Vision-Language Models (VLMs) has remained at the forefront of research discourse. We posit that this behavior stems from a reporting bias in their training data. That is, how people communicate about visual content by default omits tacit information needed to supervise some types of reasoning; e.g., "at the game today!" is a more likely caption than "a photo of 37 people standing behind a field". We investigate the data underlying the popular VLMs OpenCLIP, LLaVA-1.5 and Molmo through the lens of theories from pragmatics, and find that reporting bias results in insufficient representation of four reasoning skills (spatial, temporal, negation, and counting), despite the corpora being of web-scale, and/or synthetically generated. With a set of curated benchmarks, we demonstrate that: (i) VLMs perform poorly on the aforementioned types of reasoning suppressed in the training data by reporting bias; (ii) contrary to popular belief, scaling data size, model size, and to multiple languages does not result in emergence of these skills by default; but, promisingly, (iii) incorporating annotations specifically collected to obtain tacit information is effective. Our findings highlight the need for more intentional training data curation methods, rather than counting on scale for emergence of reasoning capabilities.