PS-GS: Gaussian Splatting for Multi-View Photometric Stereo

TL;DR

This paper introduces PS-GS, a novel method combining Gaussian splatting and inverse rendering to achieve accurate, efficient 3D reconstruction, relighting, and editing from multi-view, multi-light images.

Contribution

The paper presents PS-GS, a new approach that jointly estimates geometry, materials, and lighting using Gaussian splatting and inverse rendering, improving accuracy and efficiency.

Findings

Outperforms prior methods in reconstruction accuracy.

Demonstrates effective relighting and editing capabilities.

Achieves higher computational efficiency.

Abstract

Integrating inverse rendering with multi-view photometric stereo (MVPS) yields more accurate 3D reconstructions than the inverse rendering approaches that rely on fixed environment illumination. However, efficient inverse rendering with MVPS remains challenging. To fill this gap, we introduce the Gaussian Splatting for Multi-view Photometric Stereo (PS-GS), which efficiently and jointly estimates the geometry, materials, and lighting of the object that is illuminated by diverse directional lights (multi-light). Our method first reconstructs a standard 2D Gaussian splatting model as the initial geometry. Based on the initialization model, it then proceeds with the deferred inverse rendering by the full rendering equation containing a lighting-computing multi-layer perceptron. During the whole optimization, we regularize the rendered normal maps by the uncalibrated photometric stereo estimated normals. We also propose the 2D Gaussian ray-tracing for single directional light to refine the incident lighting. The regularizations and the use of multi-view and multi-light images mitigate the ill-posed problem of inverse rendering. After optimization, the reconstructed object can be used for novel-view synthesis, relighting, and material and shape editing. Experiments on both synthetic and real datasets demonstrate that our method outperforms prior works in terms of reconstruction accuracy and computational efficiency.