SRNN: Spatiotemporal Relational Neural Network for Intuitive Physics Understanding

TL;DR

This paper introduces SRNN, a brain-inspired neural network that models object relations and dynamics for intuitive physics understanding, bridging perception and language, and demonstrating competitive performance on the CLEVRER benchmark.

Contribution

The paper presents SRNN, a novel spatiotemporal relational neural network based on Hebbian learning principles, unifying perception and language for intuitive physics modeling.

Findings

Achieves competitive results on CLEVRER benchmark.

Reveals a bias in current evaluation methods.

Enables precise error analysis through its white-box design.

Abstract

Human prowess in intuitive physics remains unmatched by machines. To bridge this gap, we argue for a fundamental shift towards brain-inspired computational principles. This paper introduces the Spatiotemporal Relational Neural Network (SRNN), a model that establishes a unified neural representation for object attributes, relations, and timeline, with computations governed by a Hebbian ``Fire Together, Wire Together'' mechanism across dedicated \textit{What} and \textit{How} pathways. This unified representation is directly used to generate structured linguistic descriptions of the visual scene, bridging perception and language within a shared neural substrate. On the CLEVRER benchmark, SRNN achieves competitive performance, thereby confirming its capability to represent essential spatiotemporal relations from the visual stream. Cognitive ablation analysis further reveals a benchmark bias, outlining a path for a more holistic evaluation. Finally, the white-box nature of SRNN enables precise pinpointing of error root causes. Our work provides a proof-of-concept that confirms the viability of translating key principles of biological intelligence into engineered systems for intuitive physics understanding in constrained environments.