ProcGen3D: Learning Neural Procedural Graph Representations for Image-to-3D Reconstruction

TL;DR

ProcGen3D introduces a neural procedural graph-based method for 3D reconstruction from images, leveraging transformer models and MCTS to produce detailed, domain-specific 3D assets that outperform existing techniques.

Contribution

The paper presents a novel graph-based procedural representation and a transformer-based generative model with MCTS-guided sampling for improved image-to-3D reconstruction.

Findings

Outperforms state-of-the-art 3D generative methods

Enables better generalization to real-world images

Effective across diverse object categories like cacti, trees, and bridges

Abstract

We introduce ProcGen3D, a new approach for 3D content creation by generating procedural graph abstractions of 3D objects, which can then be decoded into rich, complex 3D assets. Inspired by the prevalent use of procedural generators in production 3D applications, we propose a sequentialized, graph-based procedural graph representation for 3D assets. We use this to learn to approximate the landscape of a procedural generator for image-based 3D reconstruction. We employ edge-based tokenization to encode the procedural graphs, and train a transformer prior to predict the next token conditioned on an input RGB image. Crucially, to enable better alignment of our generated outputs to an input image, we incorporate Monte Carlo Tree Search (MCTS) guided sampling into our generation process, steering output procedural graphs towards more image-faithful reconstructions. Our approach is applicable across a variety of objects that can be synthesized with procedural generators. Extensive experiments on cacti, trees, and bridges show that our neural procedural graph generation outperforms both state-of-the-art generative 3D methods and domain-specific modeling techniques. Furthermore, this enables improved generalization on real-world input images, despite training only on synthetic data.