IntPhys 2: Benchmarking Intuitive Physics Understanding In Complex Synthetic Environments

TL;DR

IntPhys 2 is a comprehensive video benchmark designed to evaluate deep learning models' understanding of core intuitive physics principles in complex synthetic environments, revealing significant gaps compared to human performance.

Contribution

The paper introduces IntPhys 2, a new benchmark focusing on four physics principles, and evaluates state-of-the-art models, highlighting their limitations in understanding intuitive physics.

Findings

Models perform at chance level on the benchmark.

Human performance is near-perfect, indicating a gap in current models.

The benchmark challenges models to differentiate possible and impossible events.

Abstract

We present IntPhys 2, a video benchmark designed to evaluate the intuitive physics understanding of deep learning models. Building on the original IntPhys benchmark, IntPhys 2 focuses on four core principles related to macroscopic objects: Permanence, Immutability, Spatio-Temporal Continuity, and Solidity. These conditions are inspired by research into intuitive physical understanding emerging during early childhood. IntPhys 2 offers a comprehensive suite of tests, based on the violation of expectation framework, that challenge models to differentiate between possible and impossible events within controlled and diverse virtual environments. Alongside the benchmark, we provide performance evaluations of several state-of-the-art models. Our findings indicate that while these models demonstrate basic visual understanding, they face significant challenges in grasping intuitive physics across the four principles in complex scenes, with most models performing at chance levels (50%), in stark contrast to human performance, which achieves near-perfect accuracy. This underscores the gap between current models and human-like intuitive physics understanding, highlighting the need for advancements in model architectures and training methodologies.