Proc3D: Procedural 3D Generation and Parametric Editing of 3D Shapes with Large Language Models

TL;DR

Proc3D introduces a novel system for generating and editing 3D models using large language models, enabling real-time, text-driven modifications with a compact graph representation that enhances efficiency and accuracy.

Contribution

The paper presents Proc3D, a system that creates editable 3D models with real-time natural language editing capabilities using a new procedural graph representation and large language models.

Findings

Achieves over 400x speedup in editing compared to traditional methods.

Improves ULIP scores by 28%, indicating better text-model alignment.

Enables intuitive, real-time parametric adjustments and text-based editing of 3D models.

Abstract

Generating 3D models has traditionally been a complex task requiring specialized expertise. While recent advances in generative AI have sought to automate this process, existing methods produce non-editable representation, such as meshes or point clouds, limiting their adaptability for iterative design. In this paper, we introduce Proc3D, a system designed to generate editable 3D models while enabling real-time modifications. At its core, Proc3D introduces procedural compact graph (PCG), a graph representation of 3D models, that encodes the algorithmic rules and structures necessary for generating the model. This representation exposes key parameters, allowing intuitive manual adjustments via sliders and checkboxes, as well as real-time, automated modifications through natural language prompts using Large Language Models (LLMs). We demonstrate Proc3D's capabilities using two generative approaches: GPT-4o with in-context learning (ICL) and a fine-tuned LLAMA-3 model. Experimental results show that Proc3D outperforms existing methods in editing efficiency, achieving more than 400x speedup over conventional approaches that require full regeneration for each modification. Additionally, Proc3D improves ULIP scores by 28%, a metric that evaluates the alignment between generated 3D models and text prompts. By enabling text-aligned 3D model generation along with precise, real-time parametric edits, Proc3D facilitates highly accurate text-based image editing applications.