Probing Physics Knowledge Using Tools from Developmental Psychology

TL;DR

This paper introduces a novel application of the violation of expectations (VOE) method from developmental psychology to evaluate and track intuitive physics understanding in artificial learning systems, using specially designed probe datasets.

Contribution

It adapts the VOE technique for AI systems, introduces new physics probe datasets inspired by developmental psychology, and demonstrates its utility in assessing physics knowledge in deep learning models.

Findings

VOE can effectively evaluate physics understanding in AI.

Baseline models show measurable responses to physics violations.

Probe datasets reveal strengths and gaps in current AI physics learning.

Abstract

In order to build agents with a rich understanding of their environment, one key objective is to endow them with a grasp of intuitive physics; an ability to reason about three-dimensional objects, their dynamic interactions, and responses to forces. While some work on this problem has taken the approach of building in components such as ready-made physics engines, other research aims to extract general physical concepts directly from sensory data. In the latter case, one challenge that arises is evaluating the learning system. Research on intuitive physics knowledge in children has long employed a violation of expectations (VOE) method to assess children's mastery of specific physical concepts. We take the novel step of applying this method to artificial learning systems. In addition to introducing the VOE technique, we describe a set of probe datasets inspired by classic test stimuli from developmental psychology. We test a baseline deep learning system on this battery, as well as on a physics learning dataset ("IntPhys") recently posed by another research group. Our results show how the VOE technique may provide a useful tool for tracking physics knowledge in future research.