Can vision language models learn intuitive physics from interaction?

TL;DR

This paper investigates whether vision-language models can develop intuitive physics through interaction, finding that current training methods, including reinforcement learning, do not produce models with generalizable physical understanding.

Contribution

It demonstrates that interaction-based training alone is insufficient for models to acquire robust, generalizable physical intuitions, highlighting a gap in current learning approaches.

Findings

Models improve within-task performance with interaction

Models fail to generalize physical knowledge to new tasks

Interaction training does not produce robust physical intuitions

Abstract

Pre-trained vision language models do not have good intuitions about the physical world. Recent work has shown that supervised fine-tuning can improve model performance on simple physical tasks. However, fine-tuned models do not appear to learn robust physical rules that can generalize to new contexts. Based on research in cognitive science, we hypothesize that models need to interact with an environment to properly learn its physical dynamics. We train models that learn through interaction with the environment using reinforcement learning. While learning from interaction allows models to improve their within-task performance, it fails to produce models with generalizable physical intuitions. We find that models trained on one task do not reliably generalize to related tasks, even if the tasks share visual statistics and physical principles, and regardless of whether the models are trained through interaction.