Learning Intuitive Physics with Multimodal Generative Models

TL;DR

This paper introduces a multimodal perception system combining visual and tactile data using a novel sensor and a variational autoencoder to predict object interactions and future states in dynamic scenes.

Contribution

It presents a new perception framework that fuses visual and tactile feedback with a novel sensor and a multimodal autoencoder for predicting object motion and interactions.

Findings

Successful prediction of object motion in simulated environments.

Effective fusion of visual and tactile data using MVAE.

Validated predictions in real-world experiments.

Abstract

Predicting the future interaction of objects when they come into contact with their environment is key for autonomous agents to take intelligent and anticipatory actions. This paper presents a perception framework that fuses visual and tactile feedback to make predictions about the expected motion of objects in dynamic scenes. Visual information captures object properties such as 3D shape and location, while tactile information provides critical cues about interaction forces and resulting object motion when it makes contact with the environment. Utilizing a novel See-Through-your-Skin (STS) sensor that provides high resolution multimodal sensing of contact surfaces, our system captures both the visual appearance and the tactile properties of objects. We interpret the dual stream signals from the sensor using a Multimodal Variational Autoencoder (MVAE), allowing us to capture both modalities of contacting objects and to develop a mapping from visual to tactile interaction and vice-versa. Additionally, the perceptual system can be used to infer the outcome of future physical interactions, which we validate through simulated and real-world experiments in which the resting state of an object is predicted from given initial conditions.