3D Mesh Editing using Masked LRMs

TL;DR

This paper introduces a fast, single-pass 3D mesh editing method using a conditional Large Reconstruction Model that can modify specific regions based on image guidance, outperforming prior methods in speed and flexibility.

Contribution

The paper proposes a novel masked reconstruction approach with a conditional LRM for 3D shape editing, enabling efficient, flexible edits from a single image.

Findings

Achieves 2-10x faster editing than prior state-of-the-art methods.

Preserves input geometry in unmasked regions with high fidelity.

Enables diverse mesh edits guided by a single image.

Abstract

We present a novel approach to shape editing, building on recent progress in 3D reconstruction from multi-view images. We formulate shape editing as a conditional reconstruction problem, where the model must reconstruct the input shape with the exception of a specified 3D region, in which the geometry should be generated from the conditional signal. To this end, we train a conditional Large Reconstruction Model (LRM) for masked reconstruction, using multi-view consistent masks rendered from a randomly generated 3D occlusion, and using one clean viewpoint as the conditional signal. During inference, we manually define a 3D region to edit and provide an edited image from a canonical viewpoint to fill that region. We demonstrate that, in just a single forward pass, our method not only preserves the input geometry in the unmasked region through reconstruction capabilities on par with SoTA, but is also expressive enough to perform a variety of mesh edits from a single image guidance that past works struggle with, while being 2-10x faster than the top-performing prior work.