SpecGen: Neural Spectral BRDF Generation via Spectral-Spatial Tri-plane Aggregation

TL;DR

SpecGen introduces a neural spectral BRDF generation method from a single RGB image, leveraging a novel spectral-spatial tri-plane network to improve spectral rendering accuracy and overcome data scarcity.

Contribution

The paper proposes the Spectral-Spatial Tri-plane Aggregation network, enabling spectral BRDF generation from limited data by leveraging RGB BRDF datasets, advancing spectral rendering techniques.

Findings

Accurately reconstructs spectral BRDFs from limited spectral data.

Surpasses state-of-the-art in hyperspectral image reconstruction with 8 dB PSNR improvement.

Effectively models reflectance across wavelengths and directions.

Abstract

Synthesizing spectral images across different wavelengths is essential for photorealistic rendering. Unlike conventional spectral uplifting methods that convert RGB images into spectral ones, we introduce SpecGen, a novel method that generates spectral bidirectional reflectance distribution functions (BRDFs) from a single RGB image of a sphere. This enables spectral image rendering under arbitrary illuminations and shapes covered by the corresponding material. A key challenge in spectral BRDF generation is the scarcity of measured spectral BRDF data. To address this, we propose the Spectral-Spatial Tri-plane Aggregation (SSTA) network, which models reflectance responses across wavelengths and incident-outgoing directions, allowing the training strategy to leverage abundant RGB BRDF data to enhance spectral BRDF generation. Experiments show that our method accurately reconstructs spectral BRDFs from limited spectral data and surpasses state-of-the-art methods in hyperspectral image reconstruction, achieving an improvement of 8 dB in PSNR. Codes and data will be released upon acceptance.