3DILG: Irregular Latent Grids for 3D Generative Modeling

TL;DR

This paper introduces 3DILG, a novel irregular grid-based neural representation for 3D shapes that enhances reconstruction accuracy and shape generation quality, advancing state-of-the-art results in probabilistic 3D modeling.

Contribution

It presents a new irregular grid-based neural field representation compatible with transformers, improving 3D shape reconstruction and generation over traditional grid-based methods.

Findings

Improved shape reconstruction accuracy from point clouds.

High-quality probabilistic shape generation from low-resolution images.

Achieved state-of-the-art results in generative 3D shape modeling.

Abstract

We propose a new representation for encoding 3D shapes as neural fields. The representation is designed to be compatible with the transformer architecture and to benefit both shape reconstruction and shape generation. Existing works on neural fields are grid-based representations with latents defined on a regular grid. In contrast, we define latents on irregular grids, enabling our representation to be sparse and adaptive. In the context of shape reconstruction from point clouds, our shape representation built on irregular grids improves upon grid-based methods in terms of reconstruction accuracy. For shape generation, our representation promotes high-quality shape generation using auto-regressive probabilistic models. We show different applications that improve over the current state of the art. First, we show results for probabilistic shape reconstruction from a single higher resolution image. Second, we train a probabilistic model conditioned on very low resolution images. Third, we apply our model to category-conditioned generation. All probabilistic experiments confirm that we are able to generate detailed and high quality shapes to yield the new state of the art in generative 3D shape modeling.