"What happens if..." Learning to Predict the Effect of Forces in Images

TL;DR

This paper introduces a deep learning approach to predict long-term object movements in images caused by external forces, utilizing a new dataset created through physics simulation of scene images.

Contribution

It presents a novel neural network model that reasons about scene geometry and object attributes to predict object movements, along with a large-scale dataset for training.

Findings

Model successfully predicts object movements from a single image.

The dataset contains over 65,000 object movements in 10,335 images.

Deep learning approach outperforms baseline methods.

Abstract

What happens if one pushes a cup sitting on a table toward the edge of the table? How about pushing a desk against a wall? In this paper, we study the problem of understanding the movements of objects as a result of applying external forces to them. For a given force vector applied to a specific location in an image, our goal is to predict long-term sequential movements caused by that force. Doing so entails reasoning about scene geometry, objects, their attributes, and the physical rules that govern the movements of objects. We design a deep neural network model that learns long-term sequential dependencies of object movements while taking into account the geometry and appearance of the scene by combining Convolutional and Recurrent Neural Networks. Training our model requires a large-scale dataset of object movements caused by external forces. To build a dataset of forces in scenes, we reconstructed all images in SUN RGB-D dataset in a physics simulator to estimate the physical movements of objects caused by external forces applied to them. Our Forces in Scenes (ForScene) dataset contains 10,335 images in which a variety of external forces are applied to different types of objects resulting in more than 65,000 object movements represented in 3D. Our experimental evaluations show that the challenging task of predicting long-term movements of objects as their reaction to external forces is possible from a single image.