NeAR: Coupled Neural Asset-Renderer Stack

TL;DR

NeAR introduces a coupled neural asset-renderer stack that jointly optimizes asset representation and rendering for improved fidelity, consistency, and real-time relighting, bridging a gap in neural graphics.

Contribution

The paper presents a novel co-designed neural asset and renderer framework, including LH-SLAT for illumination-invariant asset encoding and a lighting-aware neural decoder for real-time relighting.

Findings

Outperforms state-of-the-art in multiple relighting tasks.

Lifts casual images into a canonical, illumination-invariant latent space.

Enables real-time relightable 3D Gaussian splats without per-object optimization.

Abstract

Neural asset authoring and neural rendering have traditionally evolved as disjoint paradigms: one generates digital assets for fixed graphics pipelines, while the other maps conventional assets to images. However, treating them as independent entities limits the potential for end-to-end optimization in fidelity and consistency. In this paper, we bridge this gap with NeAR, a Coupled Neural Asset--Renderer Stack. We argue that co-designing the asset representation and the renderer creates a robust "contract" for superior generation. On the asset side, we introduce the Lighting-Homogenized SLAT (LH-SLAT). Leveraging a rectified-flow model, NeAR lifts casually lit single images into a canonical, illumination-invariant latent space, effectively suppressing baked-in shadows and highlights. On the renderer side, we design a lighting-aware neural decoder tailored to interpret these homogenized latents. Conditioned on HDR environment maps and camera views, it synthesizes relightable 3D Gaussian splats in real-time without per-object optimization. We validate NeAR on four tasks: (1) G-buffer-based forward rendering, (2) random-lit reconstruction, (3) unknown-lit relighting, and (4) novel-view relighting. Extensive experiments demonstrate that our coupled stack outperforms state-of-the-art baselines in both quantitative metrics and perceptual quality. We hope this coupled asset-renderer perspective inspires future graphics stacks that view neural assets and renderers as co-designed components instead of independent entities.