Breathing New Life into 3D Assets with Generative Repainting

TL;DR

This paper introduces a modular pipeline that combines pretrained 2D diffusion models with 3D neural radiance fields to repaint and enhance 3D assets, enabling flexible and high-quality texturing of 3D objects.

Contribution

It demonstrates a non-learned, modular approach that integrates 2D diffusion models with 3D neural fields for repainting 3D assets, allowing easy upgrades and broad applicability.

Findings

Effective repainting of 3D assets across various categories

Qualitative and quantitative improvements demonstrated

Flexible pipeline compatible with legacy geometries

Abstract

Diffusion-based text-to-image models ignited immense attention from the vision community, artists, and content creators. Broad adoption of these models is due to significant improvement in the quality of generations and efficient conditioning on various modalities, not just text. However, lifting the rich generative priors of these 2D models into 3D is challenging. Recent works have proposed various pipelines powered by the entanglement of diffusion models and neural fields. We explore the power of pretrained 2D diffusion models and standard 3D neural radiance fields as independent, standalone tools and demonstrate their ability to work together in a non-learned fashion. Such modularity has the intrinsic advantage of eased partial upgrades, which became an important property in such a fast-paced domain. Our pipeline accepts any legacy renderable geometry, such as textured or untextured meshes, orchestrates the interaction between 2D generative refinement and 3D consistency enforcement tools, and outputs a painted input geometry in several formats. We conduct a large-scale study on a wide range of objects and categories from the ShapeNetSem dataset and demonstrate the advantages of our approach, both qualitatively and quantitatively. Project page: https://www.obukhov.ai/repainting_3d_assets