Adding Intuitive Physics to Neural-Symbolic Capsules Using Interaction Networks

TL;DR

This paper introduces a neural-symbolic capsule approach that infers semantic scene information from raw images to enable learning intuitive physics directly from image sequences, improving scene understanding and physics prediction.

Contribution

It combines neural-symbolic capsules with interaction networks to infer scene semantics from pixels, advancing the ability to learn physics directly from images.

Findings

Successfully infers scene semantics from raw images

Learns intuitive physics directly from image sequences

Applies knowledge to new scenes and objects

Abstract

Many current methods to learn intuitive physics are based on interaction networks and similar approaches. However, they rely on information that has proven difficult to estimate directly from image data in the past. We aim to narrow this gap by inferring all the semantic information needed from raw pixel data in the form of a scene-graph. Our approach is based on neural-symbolic capsules, which identify which objects in the scene are static, dynamic, elastic or rigid, possible joints between them, as well as their collision information. By integrating all this with interaction networks, we demonstrate how our method is able to learn intuitive physics directly from image sequences and apply its knowledge to new scenes and objects, resulting in an inverse-simulation pipeline.