MeshReGen: A Unified 3D Geometry Regeneration Framework

TL;DR

MeshReGen is a novel 3D shape regeneration framework conditioned on initial shapes, enabling high-quality enhancement, reconstruction, and editing with state-of-the-art results using a new VecSet conditioning mechanism.

Contribution

It introduces MeshReGen, a 3D regenerator conditioned on initial shapes, utilizing VecSet for detailed updates, trained with self-supervised learning from existing datasets.

Findings

Achieves state-of-the-art performance in controllable 3D generation tasks.

Supports multiple tasks including enhancement, reconstruction, and editing.

Uses a new VecSet conditioning mechanism for fine-grained updates.

Abstract

We consider the problem of regenerating 3D objects from 2D images and initial 3D shapes. Most 3D generators operate in a one-shot fashion, converting text or images to a 3D object with limited controllability. We introduce instead MeshReGen, a 3D regenerator that is conditioned on an initial 3D shape. This conceptually simple formulation allows us to support numerous useful tasks, including 3D enhancement, reconstruction, and editing. MeshReGen uses a new conditioning mechanism based on VecSet, which allows the regenerator to update or improve the input geometry with consistent fine-grained details. MeshReGen learns a widely applicable regeneration prior from off-the-shelf 3D datasets via self-supervised pretext tasks and augmentations, without additional annotations. We evaluate both the geometric consistency and fine-grained quality of MeshReGen, achieving state-of-the-art performance in controllable 3D generation across several tasks.