PIP: Physical Interaction Prediction via Mental Simulation with Span Selection

TL;DR

This paper introduces PIP, a novel model that predicts outcomes of complex physical interactions by simulating future frames and selectively focusing on key moments, outperforming existing models and humans.

Contribution

PIP is the first model to combine mental simulation with span selection for long, multi-object physical interaction prediction, enhancing accuracy and interpretability.

Findings

PIP outperforms baseline and human predictions on the SPACE+ dataset.

Span selection effectively identifies key interaction frames.

PIP achieves superior accuracy in long sequence physical predictions.

Abstract

Accurate prediction of physical interaction outcomes is a crucial component of human intelligence and is important for safe and efficient deployments of robots in the real world. While there are existing vision-based intuitive physics models that learn to predict physical interaction outcomes, they mostly focus on generating short sequences of future frames based on physical properties (e.g. mass, friction and velocity) extracted from visual inputs or a latent space. However, there is a lack of intuitive physics models that are tested on long physical interaction sequences with multiple interactions among different objects. We hypothesize that selective temporal attention during approximate mental simulations helps humans in physical interaction outcome prediction. With these motivations, we propose a novel scheme: Physical Interaction Prediction via Mental Simulation with Span Selection (PIP). It utilizes a deep generative model to model approximate mental simulations by generating future frames of physical interactions before employing selective temporal attention in the form of span selection for predicting physical interaction outcomes. To evaluate our model, we further propose the large-scale SPACE+ dataset of synthetic videos with long sequences of three prime physical interactions in a 3D environment. Our experiments show that PIP outperforms human, baseline, and related intuitive physics models that utilize mental simulation. Furthermore, PIP's span selection module effectively identifies the frames indicating key physical interactions among objects, allowing for added interpretability.