RGM: Reconstructing High-fidelity 3D Car Assets with Relightable 3D-GS Generative Model from a Single Image

TL;DR

This paper presents a novel framework for generating high-fidelity, relightable 3D car assets from a single image, enabling realistic rendering under various lighting conditions for applications like gaming and autonomous driving.

Contribution

It introduces a relightable 3D generative model using global illumination and 3D Gaussian primitives, trained on a large synthetic dataset, to reconstruct geometry, texture, and material properties from a single image.

Findings

Produces photorealistic 3D car assets with relighting capabilities

Achieves accurate geometry, texture, and material reconstruction from a single image

Enables seamless integration into scenes with different lighting conditions

Abstract

The generation of high-quality 3D car assets is essential for various applications, including video games, autonomous driving, and virtual reality. Current 3D generation methods utilizing NeRF or 3D-GS as representations for 3D objects, generate a Lambertian object under fixed lighting and lack separated modelings for material and global illumination. As a result, the generated assets are unsuitable for relighting under varying lighting conditions, limiting their applicability in downstream tasks. To address this challenge, we propose a novel relightable 3D object generative framework that automates the creation of 3D car assets, enabling the swift and accurate reconstruction of a vehicle's geometry, texture, and material properties from a single input image. Our approach begins with introducing a large-scale synthetic car dataset comprising over 1,000 high-precision 3D vehicle models. We represent 3D objects using global illumination and relightable 3D Gaussian primitives integrating with BRDF parameters. Building on this representation, we introduce a feed-forward model that takes images as input and outputs both relightable 3D Gaussians and global illumination parameters. Experimental results demonstrate that our method produces photorealistic 3D car assets that can be seamlessly integrated into road scenes with different illuminations, which offers substantial practical benefits for industrial applications.