SE3-Nets: Learning Rigid Body Motion using Deep Neural Networks

TL;DR

SE3-Nets are deep neural networks that learn to predict rigid body motions from raw point cloud data by segmenting objects and estimating their transformations, outperforming traditional flow methods in simulation and real-world scenarios.

Contribution

This paper introduces SE3-Nets, a novel neural network architecture that directly predicts rigid body transformations from point cloud sequences, enabling more consistent motion modeling.

Findings

SE3-Nets outperform traditional flow-based networks in simulated data.

SE3-Nets accurately predict object motion in real-world robot experiments.

The model effectively segments and estimates motions of object parts from raw point clouds.

Abstract

We introduce SE3-Nets, which are deep neural networks designed to model and learn rigid body motion from raw point cloud data. Based only on sequences of depth images along with action vectors and point wise data associations, SE3-Nets learn to segment effected object parts and predict their motion resulting from the applied force. Rather than learning point wise flow vectors, SE3-Nets predict SE3 transformations for different parts of the scene. Using simulated depth data of a table top scene and a robot manipulator, we show that the structure underlying SE3-Nets enables them to generate a far more consistent prediction of object motion than traditional flow based networks. Additional experiments with a depth camera observing a Baxter robot pushing objects on a table show that SE3-Nets also work well on real data.