A Benchmark for Modeling Violation-of-Expectation in Physical Reasoning Across Event Categories

TL;DR

This paper introduces a large-scale synthetic 3D VoE dataset with ground-truth heuristics for physical reasoning, benchmarks human performance, and proposes OFPR-Net, a model that outperforms baselines by leveraging high-level causal features.

Contribution

The paper creates a novel dataset with causal heuristics for physical reasoning and develops OFPR-Net, a model that utilizes these heuristics to improve interpretability and generalize causal understanding.

Findings

OFPR-Net outperforms baseline models on the dataset.

Human performance benchmarks validate the dataset's relevance.

The dataset enables studying high-level causal reasoning in physical scenes.

Abstract

Recent work in computer vision and cognitive reasoning has given rise to an increasing adoption of the Violation-of-Expectation (VoE) paradigm in synthetic datasets. Inspired by infant psychology, researchers are now evaluating a model's ability to label scenes as either expected or surprising with knowledge of only expected scenes. However, existing VoE-based 3D datasets in physical reasoning provide mainly vision data with little to no heuristics or inductive biases. Cognitive models of physical reasoning reveal infants create high-level abstract representations of objects and interactions. Capitalizing on this knowledge, we established a benchmark to study physical reasoning by curating a novel large-scale synthetic 3D VoE dataset armed with ground-truth heuristic labels of causally relevant features and rules. To validate our dataset in five event categories of physical reasoning, we benchmarked and analyzed human performance. We also proposed the Object File Physical Reasoning Network (OFPR-Net) which exploits the dataset's novel heuristics to outperform our baseline and ablation models. The OFPR-Net is also flexible in learning an alternate physical reality, showcasing its ability to learn universal causal relationships in physical reasoning to create systems with better interpretability.