Generative VoxelNet: Learning Energy-Based Models for 3D Shape Synthesis and Analysis

TL;DR

This paper introduces a novel energy-based 3D generative model for shape synthesis and analysis, capable of producing realistic 3D objects without auxiliary models, and useful for various 3D tasks.

Contribution

The paper presents a deep energy-based model for 3D shape generation that does not rely on GANs or VAEs, enabling high-quality synthesis and versatile 3D data analysis.

Findings

The model can generate realistic 3D shapes using MCMC.

It can be applied to 3D object recovery and super-resolution.

The model serves as a powerful feature extractor for 3D classification.

Abstract

3D data that contains rich geometry information of objects and scenes is valuable for understanding 3D physical world. With the recent emergence of large-scale 3D datasets, it becomes increasingly crucial to have a powerful 3D generative model for 3D shape synthesis and analysis. This paper proposes a deep 3D energy-based model to represent volumetric shapes. The maximum likelihood training of the model follows an "analysis by synthesis" scheme. The benefits of the proposed model are six-fold: first, unlike GANs and VAEs, the model training does not rely on any auxiliary models; second, the model can synthesize realistic 3D shapes by Markov chain Monte Carlo (MCMC); third, the conditional model can be applied to 3D object recovery and super resolution; fourth, the model can serve as a building block in a multi-grid modeling and sampling framework for high resolution 3D shape synthesis; fifth, the model can be used to train a 3D generator via MCMC teaching; sixth, the unsupervisedly trained model provides a powerful feature extractor for 3D data, which is useful for 3D object classification. Experiments demonstrate that the proposed model can generate high-quality 3D shape patterns and can be useful for a wide variety of 3D shape analysis.