VIPHY: Probing "Visible" Physical Commonsense Knowledge

TL;DR

This paper evaluates vision-language models' ability to acquire and generalize visible physical commonsense knowledge related to object color, size, and space, revealing significant gaps compared to human performance.

Contribution

It introduces a comprehensive probing pipeline and dataset for assessing physical commonsense knowledge in vision-language models, highlighting their limitations.

Findings

Models perform significantly worse than humans on physical knowledge tasks.

Caption pretrained models outperform VLMs on size and spatial reasoning.

There is a notable gap between model and human understanding of visible physical knowledge.

Abstract

In recent years, vision-language models (VLMs) have shown remarkable performance on visual reasoning tasks (e.g. attributes, location). While such tasks measure the requisite knowledge to ground and reason over a given visual instance, they do not, however, measure the ability of VLMs to retain and generalize such knowledge. In this work, we evaluate their ability to acquire "visible" physical knowledge -- the information that is easily accessible from images of static scenes, particularly across the dimensions of object color, size and space. We build an automatic pipeline to derive a comprehensive knowledge resource for calibrating and probing these models. Our results indicate a severe gap between model and human performance across all three tasks. Furthermore, our caption pretrained baseline (CapBERT) significantly outperforms VLMs on both size and spatial tasks -- highlighting that despite sufficient access to ground language with visual modality, they struggle to retain such knowledge. The dataset and code are available at https://github.com/Axe--/ViPhy .