Generative PointNet: Deep Energy-Based Learning on Unordered Point Sets for 3D Generation, Reconstruction and Classification

TL;DR

This paper introduces Generative PointNet, an energy-based model for unordered point sets that learns to generate, reconstruct, and classify 3D point clouds without relying on hand-crafted metrics or auxiliary networks.

Contribution

It presents a novel energy-based generative model derived from PointNet, trained via MCMC, capable of point cloud generation, reconstruction, and classification without auxiliary networks.

Findings

Effective point cloud generation without hand-crafted metrics

Successful reconstruction and interpolation of 3D point clouds

Improved classification performance using learned representations

Abstract

We propose a generative model of unordered point sets, such as point clouds, in the form of an energy-based model, where the energy function is parameterized by an input-permutation-invariant bottom-up neural network. The energy function learns a coordinate encoding of each point and then aggregates all individual point features into an energy for the whole point cloud. We call our model the Generative PointNet because it can be derived from the discriminative PointNet. Our model can be trained by MCMC-based maximum likelihood learning (as well as its variants), without the help of any assisting networks like those in GANs and VAEs. Unlike most point cloud generators that rely on hand-crafted distance metrics, our model does not require any hand-crafted distance metric for the point cloud generation, because it synthesizes point clouds by matching observed examples in terms of statistical properties defined by the energy function. Furthermore, we can learn a short-run MCMC toward the energy-based model as a flow-like generator for point cloud reconstruction and interpolation. The learned point cloud representation can be useful for point cloud classification. Experiments demonstrate the advantages of the proposed generative model of point clouds.