Autoregressive 3D Shape Generation via Canonical Mapping

TL;DR

This paper introduces a novel transformer-based method for 3D point cloud generation by decomposing shapes into canonical sequences and using a learned codebook, achieving competitive results and enabling multi-modal shape completion.

Contribution

It proposes a new approach to 3D shape generation using canonical mapping and sequence modeling with transformers, addressing challenges in high-resolution point cloud generation.

Findings

Outperforms state-of-the-art methods in point cloud reconstruction and generation

Enables effective multi-modal shape completion

Demonstrates the feasibility of canonical sequence decomposition for 3D data

Abstract

With the capacity of modeling long-range dependencies in sequential data, transformers have shown remarkable performances in a variety of generative tasks such as image, audio, and text generation. Yet, taming them in generating less structured and voluminous data formats such as high-resolution point clouds have seldom been explored due to ambiguous sequentialization processes and infeasible computation burden. In this paper, we aim to further exploit the power of transformers and employ them for the task of 3D point cloud generation. The key idea is to decompose point clouds of one category into semantically aligned sequences of shape compositions, via a learned canonical space. These shape compositions can then be quantized and used to learn a context-rich composition codebook for point cloud generation. Experimental results on point cloud reconstruction and unconditional generation show that our model performs favorably against state-of-the-art approaches. Furthermore, our model can be easily extended to multi-modal shape completion as an application for conditional shape generation.