RPM-Net: Recurrent Prediction of Motion and Parts from Point Cloud

TL;DR

RPM-Net is a deep learning model that predicts movable parts and their motions from a single 3D point cloud, enabling hierarchical segmentation and mobility estimation of objects.

Contribution

It introduces a novel recurrent neural network architecture that jointly infers part motions and segmentations from unsegmented point clouds, including hierarchical and mobility predictions.

Findings

Effective motion prediction on synthetic and real scans.

Hierarchical segmentation of object parts based on predicted motions.

Accurate estimation of per-part mobility parameters.

Abstract

We introduce RPM-Net, a deep learning-based approach which simultaneously infers movable parts and hallucinates their motions from a single, un-segmented, and possibly partial, 3D point cloud shape. RPM-Net is a novel Recurrent Neural Network (RNN), composed of an encoder-decoder pair with interleaved Long Short-Term Memory (LSTM) components, which together predict a temporal sequence of pointwise displacements for the input point cloud. At the same time, the displacements allow the network to learn movable parts, resulting in a motion-based shape segmentation. Recursive applications of RPM-Net on the obtained parts can predict finer-level part motions, resulting in a hierarchical object segmentation. Furthermore, we develop a separate network to estimate part mobilities, e.g., per-part motion parameters, from the segmented motion sequence. Both networks learn deep predictive models from a training set that exemplifies a variety of mobilities for diverse objects. We show results of simultaneous motion and part predictions from synthetic and real scans of 3D objects exhibiting a variety of part mobilities, possibly involving multiple movable parts.