RelitLRM: Generative Relightable Radiance for Large Reconstruction Models

TL;DR

RelitLRM introduces a fast, feed-forward transformer-based model for relighting 3D object reconstructions from sparse images, outperforming traditional dense optimization methods in speed while maintaining high-quality results.

Contribution

It presents a novel large reconstruction model that decomposes geometry and appearance for relighting, trained end-to-end on synthetic data, enabling efficient and accurate relighting from sparse views.

Findings

Achieves competitive relighting quality with fewer images.

Significantly faster than dense optimization baselines.

Effectively decomposes geometry and appearance.

Abstract

We propose RelitLRM, a Large Reconstruction Model (LRM) for generating high-quality Gaussian splatting representations of 3D objects under novel illuminations from sparse (4-8) posed images captured under unknown static lighting. Unlike prior inverse rendering methods requiring dense captures and slow optimization, often causing artifacts like incorrect highlights or shadow baking, RelitLRM adopts a feed-forward transformer-based model with a novel combination of a geometry reconstructor and a relightable appearance generator based on diffusion. The model is trained end-to-end on synthetic multi-view renderings of objects under varying known illuminations. This architecture design enables to effectively decompose geometry and appearance, resolve the ambiguity between material and lighting, and capture the multi-modal distribution of shadows and specularity in the relit appearance. We show our sparse-view feed-forward RelitLRM offers competitive relighting results to state-of-the-art dense-view optimization-based baselines while being significantly faster. Our project page is available at: https://relit-lrm.github.io/.