GS-ProCams: Gaussian Splatting-based Projector-Camera Systems

TL;DR

GS-ProCams introduces a Gaussian splatting-based framework for projector-camera systems that is view-agnostic, more efficient, and requires less computational resources than previous NeRF-based methods, enabling superior projection mapping.

Contribution

It is the first Gaussian splatting-based framework for ProCams, enabling efficient, view-agnostic projection mapping without extra devices and with significantly reduced computational costs.

Findings

Achieves superior projection mapping quality compared to NeRF-based methods.

Uses only 1/10 of GPU memory for training.

Is 900 times faster in inference speed.

Abstract

We present GS-ProCams, the first Gaussian Splatting-based framework for projector-camera systems (ProCams). GS-ProCams is not only view-agnostic but also significantly enhances the efficiency of projection mapping (PM) that requires establishing geometric and radiometric mappings between the projector and the camera. Previous CNN-based ProCams are constrained to a specific viewpoint, limiting their applicability to novel perspectives. In contrast, NeRF-based ProCams support view-agnostic projection mapping, however, they require an additional co-located light source and demand significant computational and memory resources. To address this issue, we propose GS-ProCams that employs 2D Gaussian for scene representations, and enables efficient view-agnostic ProCams applications. In particular, we explicitly model the complex geometric and photometric mappings of ProCams using projector responses, the projection surface's geometry and materials represented by Gaussians, and the global illumination component. Then, we employ differentiable physically-based rendering to jointly estimate them from captured multi-view projections. Compared to state-of-the-art NeRF-based methods, our GS-ProCams eliminates the need for additional devices, achieving superior ProCams simulation quality. It also uses only 1/10 of the GPU memory for training and is 900 times faster in inference speed. Please refer to our project page for the code and dataset: https://realqingyue.github.io/GS-ProCams/.